2024-04-06T02:29:56.855 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-06T02:29:56.855 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641871 --description orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --name teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi --verbose --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641871/orig.config.yaml 2024-04-06T02:29:56.909 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T02:29:57.005 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641871 branch: squid description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7641871' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/v18.2.1 ' name: teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN sha1: 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 584a67ccb9eb224c5312dccd96a82e67b0645518 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 5609 sha1: 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 sleep_before_teardown: 0 subset: 32/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 584a67ccb9eb224c5312dccd96a82e67b0645518 targets: smithi105.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEnjEFQecBD4YhMZ8ECB/t6x08y1pkUu7IBFqwpC+gEC03+dZViMj512rGlYNPFNtbHGpo7SJr0HmMGsVW7ofq8= smithi184.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA3MvAVm7agvBv8Rfka6x0uG9XpxK6I5pmaz0CfxKt2UPPrNCDKEyzauHb95RoM0tF1IyLwDyntySDo7GwNpQn0= tasks: - install: exclude_packages: - ceph-volume tag: v18.2.1 - print: '**** done install task...' - cephadm: compiled_cephadm_branch: reef conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.io/ceph/ceph:v18.2.1 roleless: true - print: '**** done end installing v18.2.1 cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-05_21:08:03 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-04-06T02:29:57.005 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa; will attempt to use it 2024-04-06T02:29:57.006 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks 2024-04-06T02:29:57.006 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-06T02:29:57.008 INFO:teuthology.task.internal:Checking packages... 2024-04-06T02:29:57.034 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9f192350ba5fe6d0a365b251a8c2af61f56a5fa6' 2024-04-06T02:29:57.034 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-06T02:29:57.034 INFO:teuthology.packaging:ref: None 2024-04-06T02:29:57.034 INFO:teuthology.packaging:tag: None 2024-04-06T02:29:57.034 INFO:teuthology.packaging:branch: squid 2024-04-06T02:29:57.034 INFO:teuthology.packaging:sha1: 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T02:29:57.035 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-06T02:29:57.361 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2027.g9f192350 2024-04-06T02:29:57.362 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-06T02:29:57.370 INFO:teuthology.task.internal:no buildpackages task found 2024-04-06T02:29:57.370 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-06T02:29:57.414 INFO:teuthology.task.internal:Saving configuration 2024-04-06T02:29:57.435 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-06T02:29:57.447 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-06T02:29:57.472 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641871', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 02:24:48.433530', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEnjEFQecBD4YhMZ8ECB/t6x08y1pkUu7IBFqwpC+gEC03+dZViMj512rGlYNPFNtbHGpo7SJr0HmMGsVW7ofq8='} 2024-04-06T02:29:57.497 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641871', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 02:24:48.432059', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA3MvAVm7agvBv8Rfka6x0uG9XpxK6I5pmaz0CfxKt2UPPrNCDKEyzauHb95RoM0tF1IyLwDyntySDo7GwNpQn0='} 2024-04-06T02:29:57.497 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-06T02:29:57.506 INFO:teuthology.task.internal:roles: ubuntu@smithi105.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-04-06T02:29:57.506 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-04-06T02:29:57.506 INFO:teuthology.run_tasks:Running task console_log... 2024-04-06T02:29:57.600 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb3e65b70d0>, signals=[15]) 2024-04-06T02:29:57.600 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-06T02:29:57.611 INFO:teuthology.task.internal:Opening connections... 2024-04-06T02:29:57.611 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi105.front.sepia.ceph.com 2024-04-06T02:29:57.614 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:29:57.709 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi184.front.sepia.ceph.com 2024-04-06T02:29:57.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:29:57.802 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-06T02:29:57.814 DEBUG:teuthology.orchestra.run.smithi105:> uname -m 2024-04-06T02:29:57.833 INFO:teuthology.orchestra.run.smithi105.stdout:x86_64 2024-04-06T02:29:57.833 DEBUG:teuthology.orchestra.run.smithi105:> cat /etc/os-release 2024-04-06T02:29:57.888 INFO:teuthology.orchestra.run.smithi105.stdout:NAME="CentOS Stream" 2024-04-06T02:29:57.888 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION="9" 2024-04-06T02:29:57.888 INFO:teuthology.orchestra.run.smithi105.stdout:ID="centos" 2024-04-06T02:29:57.888 INFO:teuthology.orchestra.run.smithi105.stdout:ID_LIKE="rhel fedora" 2024-04-06T02:29:57.888 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION_ID="9" 2024-04-06T02:29:57.888 INFO:teuthology.orchestra.run.smithi105.stdout:PLATFORM_ID="platform:el9" 2024-04-06T02:29:57.889 INFO:teuthology.orchestra.run.smithi105.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-06T02:29:57.889 INFO:teuthology.orchestra.run.smithi105.stdout:ANSI_COLOR="0;31" 2024-04-06T02:29:57.889 INFO:teuthology.orchestra.run.smithi105.stdout:LOGO="fedora-logo-icon" 2024-04-06T02:29:57.889 INFO:teuthology.orchestra.run.smithi105.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-06T02:29:57.889 INFO:teuthology.orchestra.run.smithi105.stdout:HOME_URL="https://centos.org/" 2024-04-06T02:29:57.889 INFO:teuthology.orchestra.run.smithi105.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-06T02:29:57.889 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-06T02:29:57.889 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-06T02:29:57.890 INFO:teuthology.lock.ops:Updating smithi105.front.sepia.ceph.com on lock server 2024-04-06T02:29:57.916 DEBUG:teuthology.orchestra.run.smithi184:> uname -m 2024-04-06T02:29:57.934 INFO:teuthology.orchestra.run.smithi184.stdout:x86_64 2024-04-06T02:29:57.934 DEBUG:teuthology.orchestra.run.smithi184:> cat /etc/os-release 2024-04-06T02:29:57.992 INFO:teuthology.orchestra.run.smithi184.stdout:NAME="CentOS Stream" 2024-04-06T02:29:57.992 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION="9" 2024-04-06T02:29:57.992 INFO:teuthology.orchestra.run.smithi184.stdout:ID="centos" 2024-04-06T02:29:57.992 INFO:teuthology.orchestra.run.smithi184.stdout:ID_LIKE="rhel fedora" 2024-04-06T02:29:57.992 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_ID="9" 2024-04-06T02:29:57.992 INFO:teuthology.orchestra.run.smithi184.stdout:PLATFORM_ID="platform:el9" 2024-04-06T02:29:57.992 INFO:teuthology.orchestra.run.smithi184.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-06T02:29:57.992 INFO:teuthology.orchestra.run.smithi184.stdout:ANSI_COLOR="0;31" 2024-04-06T02:29:57.993 INFO:teuthology.orchestra.run.smithi184.stdout:LOGO="fedora-logo-icon" 2024-04-06T02:29:57.993 INFO:teuthology.orchestra.run.smithi184.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-06T02:29:57.993 INFO:teuthology.orchestra.run.smithi184.stdout:HOME_URL="https://centos.org/" 2024-04-06T02:29:57.993 INFO:teuthology.orchestra.run.smithi184.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-06T02:29:57.993 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-06T02:29:57.993 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-06T02:29:57.993 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2024-04-06T02:29:58.015 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-06T02:29:58.035 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-06T02:29:58.064 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-06T02:29:58.064 DEBUG:teuthology.orchestra.run.smithi105:> test '!' -e /home/ubuntu/cephtest 2024-04-06T02:29:58.066 DEBUG:teuthology.orchestra.run.smithi184:> test '!' -e /home/ubuntu/cephtest 2024-04-06T02:29:58.082 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-06T02:29:58.094 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-06T02:29:58.095 DEBUG:teuthology.orchestra.run.smithi105:> test -z $(ls -A /var/lib/ceph) 2024-04-06T02:29:58.122 DEBUG:teuthology.orchestra.run.smithi184:> test -z $(ls -A /var/lib/ceph) 2024-04-06T02:29:58.212 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-06T02:29:58.269 INFO:teuthology.run_tasks:Running task kernel... 2024-04-06T02:29:58.288 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-06T02:29:58.288 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-06T02:29:58.288 DEBUG:teuthology.orchestra.run.smithi105:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-06T02:29:58.289 DEBUG:teuthology.orchestra.run.smithi184:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-06T02:29:58.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T02:29:58.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T02:29:58.306 DEBUG:teuthology.orchestra.run.smithi105:> uname -r 2024-04-06T02:29:58.307 DEBUG:teuthology.orchestra.run.smithi184:> uname -r 2024-04-06T02:29:58.361 INFO:teuthology.orchestra.run.smithi105.stdout:5.14.0-361.el9.x86_64 2024-04-06T02:29:58.361 INFO:teuthology.task.kernel:Running kernel on smithi105: 5.14.0-361.el9.x86_64 2024-04-06T02:29:58.361 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum install -y kernel 2024-04-06T02:29:58.363 INFO:teuthology.orchestra.run.smithi184.stdout:5.14.0-361.el9.x86_64 2024-04-06T02:29:58.363 INFO:teuthology.task.kernel:Running kernel on smithi184: 5.14.0-361.el9.x86_64 2024-04-06T02:29:58.364 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum install -y kernel 2024-04-06T02:30:00.519 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS-9 - AppStream 17 MB/s | 8.1 MB 00:00 2024-04-06T02:30:00.623 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS-9 - AppStream 14 MB/s | 8.1 MB 00:00 2024-04-06T02:30:02.186 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.5 MB 00:00 2024-04-06T02:30:02.298 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.5 MB 00:00 2024-04-06T02:30:03.252 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-06T02:30:03.997 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - BaseOS 6.5 MB/s | 8.0 MB 00:01 2024-04-06T02:30:07.769 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - AppStream 5.5 MB/s | 19 MB 00:03 2024-04-06T02:30:09.006 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - AppStream 4.8 MB/s | 19 MB 00:03 2024-04-06T02:30:11.711 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-04-06T02:30:12.998 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-04-06T02:30:13.321 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-04-06T02:30:14.376 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 16 kB 00:00 2024-04-06T02:30:14.933 INFO:teuthology.orchestra.run.smithi184.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-06T02:30:16.626 INFO:teuthology.orchestra.run.smithi105.stdout:Extra Packages for Enterprise Linux 9.8 MB/s | 21 MB 00:02 2024-04-06T02:30:20.072 INFO:teuthology.orchestra.run.smithi184.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-04-06T02:30:21.692 INFO:teuthology.orchestra.run.smithi105.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-06T02:30:22.186 INFO:teuthology.orchestra.run.smithi184.stderr:Modular dependency problems: 2024-04-06T02:30:22.186 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-04-06T02:30:22.186 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 1: conflicting requests 2024-04-06T02:30:22.186 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.186 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 2: conflicting requests 2024-04-06T02:30:22.186 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.186 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 3: conflicting requests 2024-04-06T02:30:22.187 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.187 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 4: conflicting requests 2024-04-06T02:30:22.187 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.187 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 5: conflicting requests 2024-04-06T02:30:22.187 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.187 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 6: conflicting requests 2024-04-06T02:30:22.187 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.187 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 7: conflicting requests 2024-04-06T02:30:22.187 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.187 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 8: conflicting requests 2024-04-06T02:30:22.187 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.187 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 9: conflicting requests 2024-04-06T02:30:22.188 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.188 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 10: conflicting requests 2024-04-06T02:30:22.188 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.188 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 11: conflicting requests 2024-04-06T02:30:22.188 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.188 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 12: conflicting requests 2024-04-06T02:30:22.188 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.188 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 13: conflicting requests 2024-04-06T02:30:22.188 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.188 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 14: conflicting requests 2024-04-06T02:30:22.188 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.188 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 15: conflicting requests 2024-04-06T02:30:22.188 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.189 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 16: conflicting requests 2024-04-06T02:30:22.189 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.189 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 17: conflicting requests 2024-04-06T02:30:22.189 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.189 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 18: conflicting requests 2024-04-06T02:30:22.189 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.189 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 19: conflicting requests 2024-04-06T02:30:22.189 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.189 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 20: conflicting requests 2024-04-06T02:30:22.189 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.189 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 21: conflicting requests 2024-04-06T02:30:22.190 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.190 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 22: conflicting requests 2024-04-06T02:30:22.190 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.190 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 23: conflicting requests 2024-04-06T02:30:22.190 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.190 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 24: conflicting requests 2024-04-06T02:30:22.190 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.190 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 25: conflicting requests 2024-04-06T02:30:22.190 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.191 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 26: conflicting requests 2024-04-06T02:30:22.191 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.191 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 27: conflicting requests 2024-04-06T02:30:22.191 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.191 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 28: conflicting requests 2024-04-06T02:30:22.191 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.191 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 29: conflicting requests 2024-04-06T02:30:22.191 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.191 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 30: conflicting requests 2024-04-06T02:30:22.191 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.191 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 31: conflicting requests 2024-04-06T02:30:22.192 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.192 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 32: conflicting requests 2024-04-06T02:30:22.192 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.192 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 33: conflicting requests 2024-04-06T02:30:22.192 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.192 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 34: conflicting requests 2024-04-06T02:30:22.192 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.192 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 35: conflicting requests 2024-04-06T02:30:22.192 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.192 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 36: conflicting requests 2024-04-06T02:30:22.193 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.193 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 37: conflicting requests 2024-04-06T02:30:22.193 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.193 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.193 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.193 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:30:22.193 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.193 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.193 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.193 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.193 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 39: conflicting requests 2024-04-06T02:30:22.193 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.193 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.193 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.193 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.194 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.194 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.194 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:30:22.194 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.194 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.194 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.194 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.194 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:30:22.194 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.194 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.194 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.194 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.195 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:30:22.195 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.195 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.195 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.195 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.195 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:30:22.195 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.195 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.195 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.195 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.195 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:30:22.195 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.195 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.195 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.195 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.196 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:30:22.196 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.196 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.196 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.196 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:30:22.579 INFO:teuthology.orchestra.run.smithi184.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-06T02:30:22.580 INFO:teuthology.orchestra.run.smithi184.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-06T02:30:22.682 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-04-06T02:30:22.693 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:30:22.694 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-04-06T02:30:22.694 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:30:22.694 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-04-06T02:30:22.694 INFO:teuthology.orchestra.run.smithi184.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-04-06T02:30:22.694 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-04-06T02:30:22.694 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-04-06T02:30:22.694 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-04-06T02:30:22.694 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-04-06T02:30:22.694 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:30:22.694 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-04-06T02:30:22.694 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:30:22.695 INFO:teuthology.orchestra.run.smithi184.stdout:Install 4 Packages 2024-04-06T02:30:22.695 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:30:22.695 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 76 M 2024-04-06T02:30:22.695 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 123 M 2024-04-06T02:30:22.695 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-04-06T02:30:23.806 INFO:teuthology.orchestra.run.smithi105.stderr:Modular dependency problems: 2024-04-06T02:30:23.807 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-06T02:30:23.807 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 1: conflicting requests 2024-04-06T02:30:23.807 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.807 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 2: conflicting requests 2024-04-06T02:30:23.807 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.807 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 3: conflicting requests 2024-04-06T02:30:23.807 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.807 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 4: conflicting requests 2024-04-06T02:30:23.807 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.807 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 5: conflicting requests 2024-04-06T02:30:23.807 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.807 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 6: conflicting requests 2024-04-06T02:30:23.807 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.808 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 7: conflicting requests 2024-04-06T02:30:23.808 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.808 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 8: conflicting requests 2024-04-06T02:30:23.808 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.808 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 9: conflicting requests 2024-04-06T02:30:23.808 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.808 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 10: conflicting requests 2024-04-06T02:30:23.808 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.808 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 11: conflicting requests 2024-04-06T02:30:23.808 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.808 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 12: conflicting requests 2024-04-06T02:30:23.808 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.808 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 13: conflicting requests 2024-04-06T02:30:23.809 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.809 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 14: conflicting requests 2024-04-06T02:30:23.809 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.809 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 15: conflicting requests 2024-04-06T02:30:23.809 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.809 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 16: conflicting requests 2024-04-06T02:30:23.809 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.809 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 17: conflicting requests 2024-04-06T02:30:23.809 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.809 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 18: conflicting requests 2024-04-06T02:30:23.809 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.809 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 19: conflicting requests 2024-04-06T02:30:23.809 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.809 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 20: conflicting requests 2024-04-06T02:30:23.810 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.810 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 21: conflicting requests 2024-04-06T02:30:23.810 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.810 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 22: conflicting requests 2024-04-06T02:30:23.810 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.810 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 23: conflicting requests 2024-04-06T02:30:23.810 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.810 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 24: conflicting requests 2024-04-06T02:30:23.810 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.810 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 25: conflicting requests 2024-04-06T02:30:23.810 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.810 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 26: conflicting requests 2024-04-06T02:30:23.810 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.811 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 27: conflicting requests 2024-04-06T02:30:23.811 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.811 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 28: conflicting requests 2024-04-06T02:30:23.811 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.811 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 29: conflicting requests 2024-04-06T02:30:23.811 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.811 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 30: conflicting requests 2024-04-06T02:30:23.811 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.811 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 31: conflicting requests 2024-04-06T02:30:23.811 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.811 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 32: conflicting requests 2024-04-06T02:30:23.811 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.811 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 33: conflicting requests 2024-04-06T02:30:23.811 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.811 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 34: conflicting requests 2024-04-06T02:30:23.812 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.812 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 35: conflicting requests 2024-04-06T02:30:23.812 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.812 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 36: conflicting requests 2024-04-06T02:30:23.812 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.812 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 37: conflicting requests 2024-04-06T02:30:23.812 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.812 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.812 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.812 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:30:23.812 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.812 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.812 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.812 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.813 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 39: conflicting requests 2024-04-06T02:30:23.813 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.813 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.813 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.813 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.813 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.813 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.813 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:30:23.813 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.813 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.813 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.813 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.813 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:30:23.813 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.814 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.814 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.814 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.814 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:30:23.814 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.814 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.814 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.814 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.814 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:30:23.814 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.814 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.814 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.814 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.815 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:30:23.815 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.815 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.815 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.815 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.815 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:30:23.815 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.815 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.815 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:30:23.815 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:30:24.040 INFO:teuthology.orchestra.run.smithi184.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 62 kB/s | 66 kB 00:01 2024-04-06T02:30:24.165 INFO:teuthology.orchestra.run.smithi105.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-06T02:30:24.166 INFO:teuthology.orchestra.run.smithi105.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-06T02:30:24.280 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-06T02:30:24.291 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:30:24.291 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2024-04-06T02:30:24.291 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:30:24.291 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-06T02:30:24.291 INFO:teuthology.orchestra.run.smithi105.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-04-06T02:30:24.291 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-04-06T02:30:24.291 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-04-06T02:30:24.291 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-04-06T02:30:24.292 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-04-06T02:30:24.292 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:30:24.292 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-06T02:30:24.292 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:30:24.292 INFO:teuthology.orchestra.run.smithi105.stdout:Install 4 Packages 2024-04-06T02:30:24.292 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:30:24.292 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 76 M 2024-04-06T02:30:24.292 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 123 M 2024-04-06T02:30:24.292 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-06T02:30:24.685 INFO:teuthology.orchestra.run.smithi105.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 186 kB/s | 66 kB 00:00 2024-04-06T02:30:40.075 INFO:teuthology.orchestra.run.smithi105.stdout:(2/4): kernel-modules-core-5.14.0-432.el9.x86_6 1.8 MB/s | 27 MB 00:15 2024-04-06T02:30:41.909 INFO:teuthology.orchestra.run.smithi105.stdout:(3/4): kernel-core-5.14.0-432.el9.x86_64.rpm 869 kB/s | 15 MB 00:17 2024-04-06T02:30:43.296 INFO:teuthology.orchestra.run.smithi184.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 752 kB/s | 15 MB 00:20 2024-04-06T02:30:45.100 INFO:teuthology.orchestra.run.smithi105.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 1.6 MB/s | 33 MB 00:20 2024-04-06T02:30:45.100 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:30:45.101 INFO:teuthology.orchestra.run.smithi105.stdout:Total 3.6 MB/s | 76 MB 00:20 2024-04-06T02:30:45.764 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-06T02:30:46.137 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-06T02:30:46.137 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-06T02:30:47.141 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-06T02:30:47.141 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-06T02:30:49.617 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-06T02:30:50.444 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-04-06T02:30:50.534 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-06T02:30:53.238 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-06T02:30:53.387 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-06T02:30:58.128 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-06T02:30:58.229 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-04-06T02:31:00.660 INFO:teuthology.orchestra.run.smithi184.stdout:(3/4): kernel-modules-core-5.14.0-432.el9.x86_6 769 kB/s | 27 MB 00:36 2024-04-06T02:31:02.974 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-04-06T02:31:07.876 INFO:teuthology.orchestra.run.smithi184.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 763 kB/s | 33 MB 00:44 2024-04-06T02:31:07.876 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:31:07.877 INFO:teuthology.orchestra.run.smithi184.stdout:Total 1.7 MB/s | 76 MB 00:45 2024-04-06T02:31:08.544 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-04-06T02:31:08.919 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-04-06T02:31:08.919 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-04-06T02:31:09.910 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-04-06T02:31:09.910 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-04-06T02:31:12.370 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-04-06T02:31:13.058 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-04-06T02:31:13.151 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-06T02:31:15.895 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-06T02:31:16.022 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-06T02:31:20.753 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-06T02:31:20.797 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-04-06T02:31:25.537 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-04-06T02:31:32.192 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-04-06T02:31:32.193 INFO:teuthology.orchestra.run.smithi105.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-06T02:31:32.193 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:31:32.237 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-04-06T02:31:32.607 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-04-06T02:31:32.608 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-04-06T02:31:32.608 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-06T02:31:32.608 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-06T02:31:33.043 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-04-06T02:31:33.043 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:31:33.043 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-06T02:31:33.043 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-5.14.0-432.el9.x86_64 2024-04-06T02:31:33.044 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-04-06T02:31:33.044 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-04-06T02:31:33.044 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-04-06T02:31:33.044 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:31:33.044 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-06T02:31:33.397 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q kernel | sort -rV | head -n 1 2024-04-06T02:31:33.544 INFO:teuthology.orchestra.run.smithi105.stdout:kernel-5.14.0-432.el9.x86_64 2024-04-06T02:31:33.544 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-04-06T02:31:33.545 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-04-06T02:31:33.545 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-06T02:31:33.545 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-06T02:31:33.545 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-04-06T02:31:33.545 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi105.front.sepia.ceph.com, path=None, version=distro) 2024-04-06T02:31:33.545 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q kernel | sort -rV | head -n 1 2024-04-06T02:31:33.624 INFO:teuthology.orchestra.run.smithi105.stdout:kernel-5.14.0-432.el9.x86_64 2024-04-06T02:31:33.624 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-04-06T02:31:33.624 DEBUG:teuthology.orchestra.run.smithi105:> sudo rpm -qi grub2-tools 2024-04-06T02:31:33.662 INFO:teuthology.orchestra.run.smithi105.stdout:Name : grub2-tools 2024-04-06T02:31:33.662 INFO:teuthology.orchestra.run.smithi105.stdout:Epoch : 1 2024-04-06T02:31:33.662 INFO:teuthology.orchestra.run.smithi105.stdout:Version : 2.06 2024-04-06T02:31:33.662 INFO:teuthology.orchestra.run.smithi105.stdout:Release : 68.el9 2024-04-06T02:31:33.662 INFO:teuthology.orchestra.run.smithi105.stdout:Architecture: x86_64 2024-04-06T02:31:33.663 INFO:teuthology.orchestra.run.smithi105.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-06T02:31:33.663 INFO:teuthology.orchestra.run.smithi105.stdout:Group : Unspecified 2024-04-06T02:31:33.663 INFO:teuthology.orchestra.run.smithi105.stdout:Size : 8264254 2024-04-06T02:31:33.663 INFO:teuthology.orchestra.run.smithi105.stdout:License : GPLv3+ 2024-04-06T02:31:33.663 INFO:teuthology.orchestra.run.smithi105.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-06T02:31:33.663 INFO:teuthology.orchestra.run.smithi105.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-06T02:31:33.664 INFO:teuthology.orchestra.run.smithi105.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-06T02:31:33.664 INFO:teuthology.orchestra.run.smithi105.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-06T02:31:33.664 INFO:teuthology.orchestra.run.smithi105.stdout:Packager : builder@centos.org 2024-04-06T02:31:33.664 INFO:teuthology.orchestra.run.smithi105.stdout:Vendor : CentOS 2024-04-06T02:31:33.664 INFO:teuthology.orchestra.run.smithi105.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-06T02:31:33.664 INFO:teuthology.orchestra.run.smithi105.stdout:Summary : Support tools for GRUB. 2024-04-06T02:31:33.665 INFO:teuthology.orchestra.run.smithi105.stdout:Description : 2024-04-06T02:31:33.665 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:31:33.665 INFO:teuthology.orchestra.run.smithi105.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-06T02:31:33.665 INFO:teuthology.orchestra.run.smithi105.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-06T02:31:33.665 INFO:teuthology.orchestra.run.smithi105.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-06T02:31:33.665 INFO:teuthology.orchestra.run.smithi105.stdout:hardware devices. 2024-04-06T02:31:33.666 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:31:33.666 INFO:teuthology.orchestra.run.smithi105.stdout:This subpackage provides tools for support of all platforms. 2024-04-06T02:31:33.667 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-06T02:31:33.667 INFO:teuthology.task.kernel:Updating grub on smithi105 to boot 5.14.0-432.el9.x86_64 2024-04-06T02:31:33.668 DEBUG:teuthology.orchestra.run.smithi105:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-06T02:31:34.261 INFO:teuthology.orchestra.run.smithi105.stderr:Generating grub configuration file ... 2024-04-06T02:31:35.656 INFO:teuthology.orchestra.run.smithi105.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-06T02:31:35.686 INFO:teuthology.orchestra.run.smithi105.stderr:done 2024-04-06T02:31:35.688 DEBUG:teuthology.orchestra.run.smithi105:> mktemp 2024-04-06T02:31:35.703 INFO:teuthology.orchestra.run.smithi105.stdout:/tmp/tmp.70Cm47y7MY 2024-04-06T02:31:35.703 DEBUG:teuthology.orchestra.run.smithi105:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.70Cm47y7MY 2024-04-06T02:31:35.772 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0666 /tmp/tmp.70Cm47y7MY 2024-04-06T02:31:35.914 DEBUG:teuthology.orchestra.remote:smithi105:/tmp/tmp.70Cm47y7MY is 6KB 2024-04-06T02:31:35.933 DEBUG:teuthology.orchestra.run.smithi105:> rm -fr /tmp/tmp.70Cm47y7MY 2024-04-06T02:31:35.949 DEBUG:teuthology.orchestra.run.smithi105:> sudo /bin/ls /boot/loader/entries || true 2024-04-06T02:31:36.016 INFO:teuthology.orchestra.run.smithi105.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-06T02:31:36.016 INFO:teuthology.orchestra.run.smithi105.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-06T02:31:36.016 INFO:teuthology.orchestra.run.smithi105.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-06T02:31:36.016 INFO:teuthology.orchestra.run.smithi105.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-04-06T02:31:36.018 DEBUG:teuthology.orchestra.run.smithi105:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-04-06T02:31:36.185 DEBUG:teuthology.orchestra.run.smithi105:> sudo shutdown -r now 2024-04-06T02:31:54.844 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-04-06T02:31:54.844 INFO:teuthology.orchestra.run.smithi184.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-06T02:31:54.844 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:31:54.888 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-04-06T02:31:55.221 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-04-06T02:31:55.221 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-04-06T02:31:55.221 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-06T02:31:55.221 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-06T02:31:55.627 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-04-06T02:31:55.628 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:31:55.628 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-04-06T02:31:55.628 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-5.14.0-432.el9.x86_64 2024-04-06T02:31:55.628 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-04-06T02:31:55.628 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-04-06T02:31:55.628 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-04-06T02:31:55.628 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:31:55.628 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-04-06T02:31:55.980 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q kernel | sort -rV | head -n 1 2024-04-06T02:31:56.074 INFO:teuthology.orchestra.run.smithi184.stdout:kernel-5.14.0-432.el9.x86_64 2024-04-06T02:31:56.074 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-04-06T02:31:56.074 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-04-06T02:31:56.075 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-06T02:31:56.075 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-06T02:31:56.075 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-04-06T02:31:56.075 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi184.front.sepia.ceph.com, path=None, version=distro) 2024-04-06T02:31:56.076 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q kernel | sort -rV | head -n 1 2024-04-06T02:31:56.159 INFO:teuthology.orchestra.run.smithi184.stdout:kernel-5.14.0-432.el9.x86_64 2024-04-06T02:31:56.160 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-04-06T02:31:56.160 DEBUG:teuthology.orchestra.run.smithi184:> sudo rpm -qi grub2-tools 2024-04-06T02:31:56.204 INFO:teuthology.orchestra.run.smithi184.stdout:Name : grub2-tools 2024-04-06T02:31:56.205 INFO:teuthology.orchestra.run.smithi184.stdout:Epoch : 1 2024-04-06T02:31:56.205 INFO:teuthology.orchestra.run.smithi184.stdout:Version : 2.06 2024-04-06T02:31:56.205 INFO:teuthology.orchestra.run.smithi184.stdout:Release : 68.el9 2024-04-06T02:31:56.205 INFO:teuthology.orchestra.run.smithi184.stdout:Architecture: x86_64 2024-04-06T02:31:56.205 INFO:teuthology.orchestra.run.smithi184.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-06T02:31:56.205 INFO:teuthology.orchestra.run.smithi184.stdout:Group : Unspecified 2024-04-06T02:31:56.205 INFO:teuthology.orchestra.run.smithi184.stdout:Size : 8264254 2024-04-06T02:31:56.205 INFO:teuthology.orchestra.run.smithi184.stdout:License : GPLv3+ 2024-04-06T02:31:56.205 INFO:teuthology.orchestra.run.smithi184.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-06T02:31:56.205 INFO:teuthology.orchestra.run.smithi184.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-06T02:31:56.205 INFO:teuthology.orchestra.run.smithi184.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-06T02:31:56.206 INFO:teuthology.orchestra.run.smithi184.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-06T02:31:56.206 INFO:teuthology.orchestra.run.smithi184.stdout:Packager : builder@centos.org 2024-04-06T02:31:56.206 INFO:teuthology.orchestra.run.smithi184.stdout:Vendor : CentOS 2024-04-06T02:31:56.206 INFO:teuthology.orchestra.run.smithi184.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-06T02:31:56.206 INFO:teuthology.orchestra.run.smithi184.stdout:Summary : Support tools for GRUB. 2024-04-06T02:31:56.206 INFO:teuthology.orchestra.run.smithi184.stdout:Description : 2024-04-06T02:31:56.206 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:31:56.206 INFO:teuthology.orchestra.run.smithi184.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-06T02:31:56.206 INFO:teuthology.orchestra.run.smithi184.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-06T02:31:56.206 INFO:teuthology.orchestra.run.smithi184.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-06T02:31:56.206 INFO:teuthology.orchestra.run.smithi184.stdout:hardware devices. 2024-04-06T02:31:56.206 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:31:56.206 INFO:teuthology.orchestra.run.smithi184.stdout:This subpackage provides tools for support of all platforms. 2024-04-06T02:31:56.207 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-06T02:31:56.207 INFO:teuthology.task.kernel:Updating grub on smithi184 to boot 5.14.0-432.el9.x86_64 2024-04-06T02:31:56.208 DEBUG:teuthology.orchestra.run.smithi184:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-06T02:31:56.835 INFO:teuthology.orchestra.run.smithi184.stderr:Generating grub configuration file ... 2024-04-06T02:31:58.138 INFO:teuthology.orchestra.run.smithi184.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-06T02:31:58.178 INFO:teuthology.orchestra.run.smithi184.stderr:done 2024-04-06T02:31:58.180 DEBUG:teuthology.orchestra.run.smithi184:> mktemp 2024-04-06T02:31:58.196 INFO:teuthology.orchestra.run.smithi184.stdout:/tmp/tmp.TemzwKSuiK 2024-04-06T02:31:58.196 DEBUG:teuthology.orchestra.run.smithi184:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.TemzwKSuiK 2024-04-06T02:31:58.265 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0666 /tmp/tmp.TemzwKSuiK 2024-04-06T02:31:58.408 DEBUG:teuthology.orchestra.remote:smithi184:/tmp/tmp.TemzwKSuiK is 6KB 2024-04-06T02:31:58.458 DEBUG:teuthology.orchestra.run.smithi184:> rm -fr /tmp/tmp.TemzwKSuiK 2024-04-06T02:31:58.472 DEBUG:teuthology.orchestra.run.smithi184:> sudo /bin/ls /boot/loader/entries || true 2024-04-06T02:31:58.537 INFO:teuthology.orchestra.run.smithi184.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-06T02:31:58.537 INFO:teuthology.orchestra.run.smithi184.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-06T02:31:58.537 INFO:teuthology.orchestra.run.smithi184.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-06T02:31:58.538 INFO:teuthology.orchestra.run.smithi184.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-04-06T02:31:58.539 DEBUG:teuthology.orchestra.run.smithi184:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-04-06T02:31:58.695 DEBUG:teuthology.orchestra.run.smithi184:> sudo shutdown -r now 2024-04-06T02:32:06.194 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-04-06T02:32:06.195 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-04-06T02:32:06.195 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:32:24.610 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-04-06T02:32:28.698 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-04-06T02:32:28.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T02:32:28.699 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:32:33.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-04-06T02:32:33.615 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:32:36.670 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-04-06T02:32:48.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-04-06T02:32:48.684 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:33:07.102 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-04-06T02:33:22.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-04-06T02:33:22.117 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:33:22.488 DEBUG:teuthology.orchestra.run.smithi105:> true 2024-04-06T02:33:22.897 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-04-06T02:33:22.897 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-432.el9.x86_64"... 2024-04-06T02:33:22.898 DEBUG:teuthology.orchestra.run.smithi105:> uname -r 2024-04-06T02:33:22.912 INFO:teuthology.orchestra.run.smithi105.stdout:5.14.0-432.el9.x86_64 2024-04-06T02:33:22.912 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-04-06T02:33:22.912 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-06T02:33:22.912 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-06T02:33:23.913 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-06T02:33:23.913 DEBUG:teuthology.orchestra.run.smithi105:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-06T02:33:23.985 INFO:teuthology.orchestra.run.smithi105.stdout:ttyS1 2024-04-06T02:33:24.022 DEBUG:teuthology.parallel:result is None 2024-04-06T02:33:28.702 DEBUG:teuthology.orchestra.remote:timed out 2024-04-06T02:33:37.703 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T02:33:37.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:33:38.146 DEBUG:teuthology.orchestra.run.smithi184:> true 2024-04-06T02:33:38.541 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T02:33:38.541 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-432.el9.x86_64"... 2024-04-06T02:33:38.542 DEBUG:teuthology.orchestra.run.smithi184:> uname -r 2024-04-06T02:33:38.556 INFO:teuthology.orchestra.run.smithi184.stdout:5.14.0-432.el9.x86_64 2024-04-06T02:33:38.556 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-04-06T02:33:38.556 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-06T02:33:38.556 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-06T02:33:39.557 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-06T02:33:39.558 DEBUG:teuthology.orchestra.run.smithi184:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-06T02:33:39.636 INFO:teuthology.orchestra.run.smithi184.stdout:ttyS1 2024-04-06T02:33:39.663 DEBUG:teuthology.parallel:result is None 2024-04-06T02:33:39.663 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-06T02:33:39.673 INFO:teuthology.task.internal:Creating test directory... 2024-04-06T02:33:39.673 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-06T02:33:39.676 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-06T02:33:39.691 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-06T02:33:39.701 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-06T02:33:39.732 INFO:teuthology.task.internal:Creating archive directory... 2024-04-06T02:33:39.732 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-06T02:33:39.735 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-06T02:33:39.951 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-06T02:33:39.959 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-06T02:33:39.959 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-06T02:33:39.962 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-06T02:33:39.997 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T02:33:40.004 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T02:33:40.011 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T02:33:40.017 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T02:33:40.019 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-06T02:33:40.028 INFO:teuthology.task.internal:Configuring sudo... 2024-04-06T02:33:40.028 DEBUG:teuthology.orchestra.run.smithi105:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-06T02:33:40.055 DEBUG:teuthology.orchestra.run.smithi184:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-06T02:33:40.091 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-06T02:33:40.101 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-06T02:33:40.101 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-06T02:33:40.126 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-06T02:33:40.149 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-06T02:33:40.223 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-06T02:33:40.313 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-06T02:33:40.354 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-06T02:33:40.399 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-06T02:33:40.399 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-06T02:33:40.466 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-06T02:33:40.508 DEBUG:teuthology.orchestra.run.smithi184:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-06T02:33:40.558 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-06T02:33:40.641 DEBUG:teuthology.orchestra.run.smithi184:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-06T02:33:40.689 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-04-06T02:33:40.689 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-06T02:33:40.759 DEBUG:teuthology.orchestra.run.smithi105:> sudo service rsyslog restart 2024-04-06T02:33:40.761 DEBUG:teuthology.orchestra.run.smithi184:> sudo service rsyslog restart 2024-04-06T02:33:40.820 INFO:teuthology.orchestra.run.smithi105.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-06T02:33:40.847 INFO:teuthology.orchestra.run.smithi184.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-06T02:33:41.207 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-06T02:33:41.216 INFO:teuthology.task.internal:Starting timer... 2024-04-06T02:33:41.216 INFO:teuthology.run_tasks:Running task pcp... 2024-04-06T02:33:41.248 INFO:teuthology.run_tasks:Running task selinux... 2024-04-06T02:33:41.276 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-06T02:33:41.277 DEBUG:teuthology.orchestra.run.smithi105:> sudo service auditd rotate 2024-04-06T02:33:41.341 INFO:teuthology.orchestra.run.smithi105.stdout:Rotating logs: 2024-04-06T02:33:41.343 DEBUG:teuthology.orchestra.run.smithi184:> sudo service auditd rotate 2024-04-06T02:33:41.406 INFO:teuthology.orchestra.run.smithi184.stdout:Rotating logs: 2024-04-06T02:33:41.408 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-06T02:33:41.408 DEBUG:teuthology.orchestra.run.smithi105:> /usr/sbin/getenforce 2024-04-06T02:33:41.432 INFO:teuthology.orchestra.run.smithi105.stdout:Permissive 2024-04-06T02:33:41.432 DEBUG:teuthology.orchestra.run.smithi184:> /usr/sbin/getenforce 2024-04-06T02:33:41.453 INFO:teuthology.orchestra.run.smithi184.stdout:Permissive 2024-04-06T02:33:41.453 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi105.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi184.front.sepia.ceph.com': 'permissive'} 2024-04-06T02:33:41.454 DEBUG:teuthology.orchestra.run.smithi105:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-06T02:33:41.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T02:33:41.501 DEBUG:teuthology.orchestra.run.smithi184:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-06T02:33:41.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T02:33:41.529 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-06T02:33:41.529 DEBUG:teuthology.orchestra.run.smithi105:> sudo /usr/sbin/setenforce permissive 2024-04-06T02:33:41.591 DEBUG:teuthology.orchestra.run.smithi184:> sudo /usr/sbin/setenforce permissive 2024-04-06T02:33:41.630 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-06T02:33:41.640 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-06T02:33:41.646 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-06T02:33:41.780 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-06T02:33:41.798 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-06T02:33:41.799 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi105.front.sepia.ceph.com,smithi184.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-06T02:38:00.952 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi105.front.sepia.ceph.com'), Remote(name='ubuntu@smithi184.front.sepia.ceph.com')] 2024-04-06T02:38:00.954 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-04-06T02:38:00.956 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:38:01.044 DEBUG:teuthology.orchestra.run.smithi105:> true 2024-04-06T02:38:01.117 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-04-06T02:38:01.117 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T02:38:01.118 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:38:01.207 DEBUG:teuthology.orchestra.run.smithi184:> true 2024-04-06T02:38:01.290 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T02:38:01.290 INFO:teuthology.run_tasks:Running task clock... 2024-04-06T02:38:01.300 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-06T02:38:01.300 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-06T02:38:01.301 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T02:38:01.303 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-06T02:38:01.303 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T02:38:01.331 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-06T02:38:01.347 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-06T02:38:01.362 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-06T02:38:01.376 INFO:teuthology.orchestra.run.smithi105.stderr:sudo: ntpd: command not found 2024-04-06T02:38:01.379 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-06T02:38:01.390 INFO:teuthology.orchestra.run.smithi105.stdout:506 Cannot talk to daemon 2024-04-06T02:38:01.406 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-06T02:38:01.409 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: ntpd: command not found 2024-04-06T02:38:01.422 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-06T02:38:01.423 INFO:teuthology.orchestra.run.smithi184.stdout:506 Cannot talk to daemon 2024-04-06T02:38:01.439 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-06T02:38:01.456 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-06T02:38:01.475 INFO:teuthology.orchestra.run.smithi105.stderr:bash: line 1: ntpq: command not found 2024-04-06T02:38:01.478 INFO:teuthology.orchestra.run.smithi105.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-06T02:38:01.478 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================== 2024-04-06T02:38:01.478 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T02:38:01.478 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T02:38:01.478 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T02:38:01.479 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T02:38:01.510 INFO:teuthology.orchestra.run.smithi184.stderr:bash: line 1: ntpq: command not found 2024-04-06T02:38:01.515 INFO:teuthology.orchestra.run.smithi184.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-06T02:38:01.515 INFO:teuthology.orchestra.run.smithi184.stdout:=============================================================================== 2024-04-06T02:38:01.515 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T02:38:01.515 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T02:38:01.515 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T02:38:01.515 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T02:38:01.515 INFO:teuthology.run_tasks:Running task install... 2024-04-06T02:38:01.524 DEBUG:teuthology.task.install:project ceph 2024-04-06T02:38:01.524 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '9f192350ba5fe6d0a365b251a8c2af61f56a5fa6'}} 2024-04-06T02:38:01.524 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v18.2.1', 'flavor': 'default', 'sha1': '9f192350ba5fe6d0a365b251a8c2af61f56a5fa6'} 2024-04-06T02:38:01.524 INFO:teuthology.task.install:Using flavor: default 2024-04-06T02:38:01.530 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-06T02:38:01.530 INFO:teuthology.task.install:extra packages: [] 2024-04-06T02:38:01.530 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9f192350ba5fe6d0a365b251a8c2af61f56a5fa6', 'tag': 'v18.2.1', 'wait_for_package': False} 2024-04-06T02:38:01.530 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-06T02:38:01.531 INFO:teuthology.packaging:ref: None 2024-04-06T02:38:01.531 INFO:teuthology.packaging:tag: v18.2.1 2024-04-06T02:38:01.531 INFO:teuthology.packaging:branch: None 2024-04-06T02:38:01.531 INFO:teuthology.packaging:sha1: 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T02:38:03.649 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph v18.2.1^{} -> 7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-04-06T02:38:03.650 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-04-06T02:38:03.652 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9f192350ba5fe6d0a365b251a8c2af61f56a5fa6', 'tag': 'v18.2.1', 'wait_for_package': False} 2024-04-06T02:38:03.653 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-06T02:38:03.653 INFO:teuthology.packaging:ref: None 2024-04-06T02:38:03.653 INFO:teuthology.packaging:tag: v18.2.1 2024-04-06T02:38:03.653 INFO:teuthology.packaging:branch: None 2024-04-06T02:38:03.653 INFO:teuthology.packaging:sha1: 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T02:38:03.653 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-04-06T02:38:03.874 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/ 2024-04-06T02:38:03.874 INFO:teuthology.task.install.rpm:Package version is 18.2.1-0 2024-04-06T02:38:03.880 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/ 2024-04-06T02:38:03.880 INFO:teuthology.task.install.rpm:Package version is 18.2.1-0 2024-04-06T02:38:03.931 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-06T02:38:03.931 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-06T02:38:03.931 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-06T02:38:03.934 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-06T02:38:03.934 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-04-06T02:38:03.934 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-06T02:38:03.964 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-06T02:38:03.964 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-06T02:38:03.964 INFO:teuthology.packaging:ref: None 2024-04-06T02:38:03.964 INFO:teuthology.packaging:tag: v18.2.1 2024-04-06T02:38:03.964 INFO:teuthology.packaging:branch: None 2024-04-06T02:38:03.964 INFO:teuthology.packaging:sha1: 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T02:38:03.964 DEBUG:teuthology.orchestra.run.smithi105:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-06T02:38:03.969 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-06T02:38:03.969 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-06T02:38:03.970 INFO:teuthology.packaging:ref: None 2024-04-06T02:38:03.970 INFO:teuthology.packaging:tag: v18.2.1 2024-04-06T02:38:03.970 INFO:teuthology.packaging:branch: None 2024-04-06T02:38:03.970 INFO:teuthology.packaging:sha1: 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T02:38:03.970 DEBUG:teuthology.orchestra.run.smithi184:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-06T02:38:04.036 DEBUG:teuthology.orchestra.run.smithi105:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-06T02:38:04.041 DEBUG:teuthology.orchestra.run.smithi184:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-06T02:38:04.121 DEBUG:teuthology.orchestra.run.smithi105:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-06T02:38:04.126 DEBUG:teuthology.orchestra.run.smithi184:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-06T02:38:04.190 INFO:teuthology.orchestra.run.smithi105.stdout:check_obsoletes = 1 2024-04-06T02:38:04.191 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum clean all 2024-04-06T02:38:04.195 INFO:teuthology.orchestra.run.smithi184.stdout:check_obsoletes = 1 2024-04-06T02:38:04.196 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum clean all 2024-04-06T02:38:04.449 INFO:teuthology.orchestra.run.smithi105.stdout:74 files removed 2024-04-06T02:38:04.450 INFO:teuthology.orchestra.run.smithi184.stdout:74 files removed 2024-04-06T02:38:04.479 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-radosgw 2024-04-06T02:38:04.481 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-radosgw 2024-04-06T02:38:04.993 INFO:teuthology.orchestra.run.smithi105.stdout:ceph packages for x86_64 360 kB/s | 76 kB 00:00 2024-04-06T02:38:05.001 INFO:teuthology.orchestra.run.smithi184.stdout:ceph packages for x86_64 347 kB/s | 76 kB 00:00 2024-04-06T02:38:05.210 INFO:teuthology.orchestra.run.smithi184.stdout:ceph noarch packages 59 kB/s | 9.4 kB 00:00 2024-04-06T02:38:05.227 INFO:teuthology.orchestra.run.smithi105.stdout:ceph noarch packages 51 kB/s | 9.4 kB 00:00 2024-04-06T02:38:05.419 INFO:teuthology.orchestra.run.smithi184.stdout:ceph source packages 13 kB/s | 2.2 kB 00:00 2024-04-06T02:38:05.436 INFO:teuthology.orchestra.run.smithi105.stdout:ceph source packages 13 kB/s | 2.2 kB 00:00 2024-04-06T02:38:05.897 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS-9 - AppStream 19 MB/s | 8.1 MB 00:00 2024-04-06T02:38:05.928 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS-9 - AppStream 17 MB/s | 8.1 MB 00:00 2024-04-06T02:38:07.547 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS-9 - BaseOS 7.9 MB/s | 2.5 MB 00:00 2024-04-06T02:38:07.555 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS-9 - BaseOS 8.1 MB/s | 2.5 MB 00:00 2024-04-06T02:38:08.749 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-06T02:38:08.888 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - BaseOS 9.1 MB/s | 8.0 MB 00:00 2024-04-06T02:38:11.033 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-04-06T02:38:11.214 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-04-06T02:38:15.215 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-04-06T02:38:16.632 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 16 kB 00:00 2024-04-06T02:38:18.181 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - CRB 1.7 MB/s | 6.1 MB 00:03 2024-04-06T02:38:19.899 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 16 kB 00:00 2024-04-06T02:38:21.036 INFO:teuthology.orchestra.run.smithi105.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:01 2024-04-06T02:38:26.067 INFO:teuthology.orchestra.run.smithi105.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-06T02:38:28.489 INFO:teuthology.orchestra.run.smithi105.stderr:Modular dependency problems: 2024-04-06T02:38:28.490 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-06T02:38:28.490 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 1: conflicting requests 2024-04-06T02:38:28.490 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.490 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 2: conflicting requests 2024-04-06T02:38:28.490 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.490 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 3: conflicting requests 2024-04-06T02:38:28.490 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.490 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 4: conflicting requests 2024-04-06T02:38:28.490 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.490 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 5: conflicting requests 2024-04-06T02:38:28.491 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.491 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 6: conflicting requests 2024-04-06T02:38:28.491 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.491 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 7: conflicting requests 2024-04-06T02:38:28.491 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.491 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 8: conflicting requests 2024-04-06T02:38:28.491 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.491 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 9: conflicting requests 2024-04-06T02:38:28.491 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.491 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 10: conflicting requests 2024-04-06T02:38:28.491 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.491 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 11: conflicting requests 2024-04-06T02:38:28.491 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.492 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 12: conflicting requests 2024-04-06T02:38:28.492 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.492 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 13: conflicting requests 2024-04-06T02:38:28.492 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.492 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 14: conflicting requests 2024-04-06T02:38:28.492 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.492 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 15: conflicting requests 2024-04-06T02:38:28.492 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.492 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 16: conflicting requests 2024-04-06T02:38:28.492 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.492 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 17: conflicting requests 2024-04-06T02:38:28.492 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.492 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 18: conflicting requests 2024-04-06T02:38:28.493 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.493 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 19: conflicting requests 2024-04-06T02:38:28.493 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.493 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 20: conflicting requests 2024-04-06T02:38:28.493 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.493 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 21: conflicting requests 2024-04-06T02:38:28.493 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.493 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 22: conflicting requests 2024-04-06T02:38:28.493 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.493 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 23: conflicting requests 2024-04-06T02:38:28.493 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.493 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 24: conflicting requests 2024-04-06T02:38:28.493 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.494 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 25: conflicting requests 2024-04-06T02:38:28.494 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.494 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 26: conflicting requests 2024-04-06T02:38:28.494 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.494 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 27: conflicting requests 2024-04-06T02:38:28.494 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.494 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 28: conflicting requests 2024-04-06T02:38:28.494 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.494 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 29: conflicting requests 2024-04-06T02:38:28.494 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.494 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 30: conflicting requests 2024-04-06T02:38:28.494 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.495 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 31: conflicting requests 2024-04-06T02:38:28.495 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.495 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 32: conflicting requests 2024-04-06T02:38:28.495 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.495 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 33: conflicting requests 2024-04-06T02:38:28.495 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.495 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 34: conflicting requests 2024-04-06T02:38:28.495 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.495 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 35: conflicting requests 2024-04-06T02:38:28.495 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.495 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 36: conflicting requests 2024-04-06T02:38:28.495 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.495 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 37: conflicting requests 2024-04-06T02:38:28.495 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.496 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.496 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.496 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:38:28.496 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.496 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.496 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.496 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.496 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 39: conflicting requests 2024-04-06T02:38:28.496 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.496 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.496 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.496 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.496 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.497 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.497 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:38:28.497 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.497 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.497 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.497 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.497 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:38:28.497 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.497 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.497 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.497 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.497 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:38:28.497 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.498 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.498 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.498 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.498 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:38:28.498 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.498 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.498 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.498 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.498 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:38:28.498 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.498 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.498 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.498 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.498 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:38:28.499 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.499 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.499 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:38:28.499 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:38:29.016 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-06T02:38:29.035 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:38:29.035 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-04-06T02:38:29.035 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:38:29.036 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-06T02:38:29.036 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw x86_64 2:18.2.1-0.el9 ceph 7.7 M 2024-04-06T02:38:29.036 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrading: 2024-04-06T02:38:29.036 INFO:teuthology.orchestra.run.smithi105.stdout: librados2 x86_64 2:18.2.1-0.el9 ceph 3.3 M 2024-04-06T02:38:29.036 INFO:teuthology.orchestra.run.smithi105.stdout: librbd1 x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-04-06T02:38:29.036 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-04-06T02:38:29.036 INFO:teuthology.orchestra.run.smithi105.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-06T02:38:29.036 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base x86_64 2:18.2.1-0.el9 ceph 5.2 M 2024-04-06T02:38:29.036 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common x86_64 2:18.2.1-0.el9 ceph 18 M 2024-04-06T02:38:29.036 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux x86_64 2:18.2.1-0.el9 ceph 24 k 2024-04-06T02:38:29.036 INFO:teuthology.orchestra.run.smithi105.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-04-06T02:38:29.036 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-06T02:38:29.036 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-06T02:38:29.037 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-06T02:38:29.037 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2 x86_64 2:18.2.1-0.el9 ceph 658 k 2024-04-06T02:38:29.037 INFO:teuthology.orchestra.run.smithi105.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-06T02:38:29.037 INFO:teuthology.orchestra.run.smithi105.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-06T02:38:29.037 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-06T02:38:29.037 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1 x86_64 2:18.2.1-0.el9 ceph 474 k 2024-04-06T02:38:29.037 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-06T02:38:29.037 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2 x86_64 2:18.2.1-0.el9 ceph 4.5 M 2024-04-06T02:38:29.037 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-06T02:38:29.037 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-06T02:38:29.037 INFO:teuthology.orchestra.run.smithi105.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-06T02:38:29.037 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse x86_64 2:18.2.1-0.el9 ceph 45 k 2024-04-06T02:38:29.037 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common x86_64 2:18.2.1-0.el9 ceph 124 k 2024-04-06T02:38:29.037 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs x86_64 2:18.2.1-0.el9 ceph 161 k 2024-04-06T02:38:29.038 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-06T02:38:29.038 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados x86_64 2:18.2.1-0.el9 ceph 321 k 2024-04-06T02:38:29.038 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd x86_64 2:18.2.1-0.el9 ceph 297 k 2024-04-06T02:38:29.038 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw x86_64 2:18.2.1-0.el9 ceph 99 k 2024-04-06T02:38:29.038 INFO:teuthology.orchestra.run.smithi105.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-06T02:38:29.038 INFO:teuthology.orchestra.run.smithi105.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-06T02:38:29.038 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:38:29.038 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-06T02:38:29.038 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:38:29.038 INFO:teuthology.orchestra.run.smithi105.stdout:Install 28 Packages 2024-04-06T02:38:29.038 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrade 2 Packages 2024-04-06T02:38:29.038 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:38:29.039 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 53 M 2024-04-06T02:38:29.039 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-06T02:38:29.627 INFO:teuthology.orchestra.run.smithi105.stdout:(1/30): ceph-base-18.2.1-0.el9.x86_64.rpm 17 MB/s | 5.2 MB 00:00 2024-04-06T02:38:29.668 INFO:teuthology.orchestra.run.smithi105.stdout:(2/30): ceph-selinux-18.2.1-0.el9.x86_64.rpm 580 kB/s | 24 kB 00:00 2024-04-06T02:38:29.719 INFO:teuthology.orchestra.run.smithi105.stdout:(3/30): libcephfs2-18.2.1-0.el9.x86_64.rpm 13 MB/s | 658 kB 00:00 2024-04-06T02:38:30.186 INFO:teuthology.orchestra.run.smithi105.stdout:(4/30): ceph-common-18.2.1-0.el9.x86_64.rpm 21 MB/s | 18 MB 00:00 2024-04-06T02:38:30.219 INFO:teuthology.orchestra.run.smithi105.stdout:(5/30): libradosstriper1-18.2.1-0.el9.x86_64.rp 948 kB/s | 474 kB 00:00 2024-04-06T02:38:30.328 INFO:teuthology.orchestra.run.smithi105.stdout:(6/30): ceph-radosgw-18.2.1-0.el9.x86_64.rpm 7.7 MB/s | 7.7 MB 00:01 2024-04-06T02:38:30.345 INFO:teuthology.orchestra.run.smithi105.stdout:(7/30): python3-ceph-argparse-18.2.1-0.el9.x86_ 358 kB/s | 45 kB 00:00 2024-04-06T02:38:30.370 INFO:teuthology.orchestra.run.smithi105.stdout:(8/30): python3-ceph-common-18.2.1-0.el9.x86_64 2.9 MB/s | 124 kB 00:00 2024-04-06T02:38:30.412 INFO:teuthology.orchestra.run.smithi105.stdout:(9/30): python3-cephfs-18.2.1-0.el9.x86_64.rpm 2.4 MB/s | 161 kB 00:00 2024-04-06T02:38:30.437 INFO:teuthology.orchestra.run.smithi105.stdout:(10/30): python3-rados-18.2.1-0.el9.x86_64.rpm 4.7 MB/s | 321 kB 00:00 2024-04-06T02:38:30.505 INFO:teuthology.orchestra.run.smithi105.stdout:(11/30): librgw2-18.2.1-0.el9.x86_64.rpm 14 MB/s | 4.5 MB 00:00 2024-04-06T02:38:30.530 INFO:teuthology.orchestra.run.smithi105.stdout:(12/30): python3-rgw-18.2.1-0.el9.x86_64.rpm 1.1 MB/s | 99 kB 00:00 2024-04-06T02:38:30.555 INFO:teuthology.orchestra.run.smithi105.stdout:(13/30): python3-rbd-18.2.1-0.el9.x86_64.rpm 2.0 MB/s | 297 kB 00:00 2024-04-06T02:38:30.705 INFO:teuthology.orchestra.run.smithi105.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 301 kB/s | 45 kB 00:00 2024-04-06T02:38:30.730 INFO:teuthology.orchestra.run.smithi105.stdout:(15/30): boost-program-options-1.75.0-8.el9.x86 474 kB/s | 107 kB 00:00 2024-04-06T02:38:30.764 INFO:teuthology.orchestra.run.smithi105.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 683 kB/s | 160 kB 00:00 2024-04-06T02:38:30.814 INFO:teuthology.orchestra.run.smithi105.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 849 kB/s | 42 kB 00:00 2024-04-06T02:38:30.856 INFO:teuthology.orchestra.run.smithi105.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-04-06T02:38:30.882 INFO:teuthology.orchestra.run.smithi105.stdout:(19/30): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 662 kB 00:00 2024-04-06T02:38:31.432 INFO:teuthology.orchestra.run.smithi105.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 179 kB/s | 111 kB 00:00 2024-04-06T02:38:31.540 INFO:teuthology.orchestra.run.smithi105.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-04-06T02:38:31.732 INFO:teuthology.orchestra.run.smithi105.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 253 kB/s | 49 kB 00:00 2024-04-06T02:38:31.758 INFO:teuthology.orchestra.run.smithi105.stdout:(23/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 343 kB/s | 309 kB 00:00 2024-04-06T02:38:31.916 INFO:teuthology.orchestra.run.smithi105.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 367 kB/s | 67 kB 00:00 2024-04-06T02:38:32.275 INFO:teuthology.orchestra.run.smithi105.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 535 kB/s | 192 kB 00:00 2024-04-06T02:38:32.850 INFO:teuthology.orchestra.run.smithi105.stdout:(26/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 767 kB/s | 838 kB 00:01 2024-04-06T02:38:32.984 INFO:teuthology.orchestra.run.smithi105.stdout:(27/30): librados2-18.2.1-0.el9.x86_64.rpm 25 MB/s | 3.3 MB 00:00 2024-04-06T02:38:33.126 INFO:teuthology.orchestra.run.smithi105.stdout:(28/30): librbd1-18.2.1-0.el9.x86_64.rpm 21 MB/s | 3.0 MB 00:00 2024-04-06T02:38:33.740 INFO:teuthology.orchestra.run.smithi184.stdout:Extra Packages for Enterprise Linux 1.2 MB/s | 21 MB 00:16 2024-04-06T02:38:34.517 INFO:teuthology.orchestra.run.smithi105.stdout:(29/30): thrift-0.15.0-2.el9.x86_64.rpm 722 kB/s | 1.6 MB 00:02 2024-04-06T02:38:36.784 INFO:teuthology.orchestra.run.smithi105.stdout:(30/30): libarrow-9.0.0-10.el9.x86_64.rpm 766 kB/s | 4.4 MB 00:05 2024-04-06T02:38:36.789 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:38:36.789 INFO:teuthology.orchestra.run.smithi105.stdout:Total 6.8 MB/s | 53 MB 00:07 2024-04-06T02:38:36.862 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-06T02:38:36.898 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-06T02:38:36.898 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-06T02:38:37.343 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-06T02:38:37.343 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-06T02:38:38.714 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-06T02:38:38.790 INFO:teuthology.orchestra.run.smithi184.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-06T02:38:38.801 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-04-06T02:38:38.839 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-04-06T02:38:38.897 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-04-06T02:38:39.211 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-04-06T02:38:39.271 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librados2-2:18.2.1-0.el9.x86_64 5/32 2024-04-06T02:38:39.381 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-2:18.2.1-0.el9.x86_64 5/32 2024-04-06T02:38:39.460 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rados-2:18.2.1-0.el9.x86_64 6/32 2024-04-06T02:38:39.498 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-04-06T02:38:39.561 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-04-06T02:38:39.589 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-04-06T02:38:39.633 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 9/32 2024-04-06T02:38:39.699 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-04-06T02:38:39.725 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-04-06T02:38:39.774 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-04-06T02:38:39.881 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-04-06T02:38:39.917 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-04-06T02:38:39.956 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-04-06T02:38:39.989 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-04-06T02:38:40.016 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-04-06T02:38:40.052 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-04-06T02:38:40.084 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-04-06T02:38:40.375 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-04-06T02:38:40.406 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-04-06T02:38:40.496 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-04-06T02:38:40.527 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rbd-2:18.2.1-0.el9.x86_64 21/32 2024-04-06T02:38:40.563 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-04-06T02:38:40.966 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-04-06T02:38:40.992 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-04-06T02:38:41.063 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-04-06T02:38:41.139 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rgw-2:18.2.1-0.el9.x86_64 25/32 2024-04-06T02:38:41.169 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-common-2:18.2.1-0.el9.x86_64 26/32 2024-04-06T02:38:41.216 INFO:teuthology.orchestra.run.smithi184.stderr:Modular dependency problems: 2024-04-06T02:38:41.217 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-04-06T02:38:41.217 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 1: conflicting requests 2024-04-06T02:38:41.217 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.217 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 2: conflicting requests 2024-04-06T02:38:41.217 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.217 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 3: conflicting requests 2024-04-06T02:38:41.217 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.217 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 4: conflicting requests 2024-04-06T02:38:41.217 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.217 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 5: conflicting requests 2024-04-06T02:38:41.217 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.217 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 6: conflicting requests 2024-04-06T02:38:41.218 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.218 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 7: conflicting requests 2024-04-06T02:38:41.218 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.218 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 8: conflicting requests 2024-04-06T02:38:41.218 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.218 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 9: conflicting requests 2024-04-06T02:38:41.218 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.218 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 10: conflicting requests 2024-04-06T02:38:41.218 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.218 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 11: conflicting requests 2024-04-06T02:38:41.218 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.218 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 12: conflicting requests 2024-04-06T02:38:41.218 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.219 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 13: conflicting requests 2024-04-06T02:38:41.219 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.219 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 14: conflicting requests 2024-04-06T02:38:41.219 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.219 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 15: conflicting requests 2024-04-06T02:38:41.219 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.219 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 16: conflicting requests 2024-04-06T02:38:41.219 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.219 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 17: conflicting requests 2024-04-06T02:38:41.219 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.219 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 18: conflicting requests 2024-04-06T02:38:41.219 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.219 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 19: conflicting requests 2024-04-06T02:38:41.220 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.220 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 20: conflicting requests 2024-04-06T02:38:41.220 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.220 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 21: conflicting requests 2024-04-06T02:38:41.220 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.220 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 22: conflicting requests 2024-04-06T02:38:41.220 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.220 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 23: conflicting requests 2024-04-06T02:38:41.220 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.220 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 24: conflicting requests 2024-04-06T02:38:41.220 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.220 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 25: conflicting requests 2024-04-06T02:38:41.220 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.221 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 26: conflicting requests 2024-04-06T02:38:41.221 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.221 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 27: conflicting requests 2024-04-06T02:38:41.221 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.221 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 28: conflicting requests 2024-04-06T02:38:41.221 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.221 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 29: conflicting requests 2024-04-06T02:38:41.221 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.221 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 30: conflicting requests 2024-04-06T02:38:41.221 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.221 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 31: conflicting requests 2024-04-06T02:38:41.221 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.221 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 32: conflicting requests 2024-04-06T02:38:41.221 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.222 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 33: conflicting requests 2024-04-06T02:38:41.222 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.222 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 34: conflicting requests 2024-04-06T02:38:41.222 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.222 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 35: conflicting requests 2024-04-06T02:38:41.222 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.222 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 36: conflicting requests 2024-04-06T02:38:41.222 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.222 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 37: conflicting requests 2024-04-06T02:38:41.222 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.222 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.222 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.222 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:38:41.222 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.223 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.223 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.223 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.223 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 39: conflicting requests 2024-04-06T02:38:41.223 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.223 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.223 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.223 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.223 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.223 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.223 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:38:41.223 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.223 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.224 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.224 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.224 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:38:41.224 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.224 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.224 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.224 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.224 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:38:41.224 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.224 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.224 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.224 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.224 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:38:41.224 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.225 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.225 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.225 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.225 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:38:41.225 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.225 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.225 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.225 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.225 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:38:41.225 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.225 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.225 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.225 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:38:41.764 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-04-06T02:38:41.774 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:38:41.774 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-04-06T02:38:41.774 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:38:41.774 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-04-06T02:38:41.774 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-radosgw x86_64 2:18.2.1-0.el9 ceph 7.7 M 2024-04-06T02:38:41.774 INFO:teuthology.orchestra.run.smithi184.stdout:Upgrading: 2024-04-06T02:38:41.774 INFO:teuthology.orchestra.run.smithi184.stdout: librados2 x86_64 2:18.2.1-0.el9 ceph 3.3 M 2024-04-06T02:38:41.774 INFO:teuthology.orchestra.run.smithi184.stdout: librbd1 x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-04-06T02:38:41.774 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-04-06T02:38:41.774 INFO:teuthology.orchestra.run.smithi184.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-06T02:38:41.775 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-base x86_64 2:18.2.1-0.el9 ceph 5.2 M 2024-04-06T02:38:41.775 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-common x86_64 2:18.2.1-0.el9 ceph 18 M 2024-04-06T02:38:41.775 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-selinux x86_64 2:18.2.1-0.el9 ceph 24 k 2024-04-06T02:38:41.775 INFO:teuthology.orchestra.run.smithi184.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-04-06T02:38:41.775 INFO:teuthology.orchestra.run.smithi184.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-06T02:38:41.775 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-06T02:38:41.775 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-06T02:38:41.775 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs2 x86_64 2:18.2.1-0.el9 ceph 658 k 2024-04-06T02:38:41.775 INFO:teuthology.orchestra.run.smithi184.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-06T02:38:41.775 INFO:teuthology.orchestra.run.smithi184.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-06T02:38:41.775 INFO:teuthology.orchestra.run.smithi184.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-06T02:38:41.775 INFO:teuthology.orchestra.run.smithi184.stdout: libradosstriper1 x86_64 2:18.2.1-0.el9 ceph 474 k 2024-04-06T02:38:41.775 INFO:teuthology.orchestra.run.smithi184.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-06T02:38:41.776 INFO:teuthology.orchestra.run.smithi184.stdout: librgw2 x86_64 2:18.2.1-0.el9 ceph 4.5 M 2024-04-06T02:38:41.776 INFO:teuthology.orchestra.run.smithi184.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-06T02:38:41.776 INFO:teuthology.orchestra.run.smithi184.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-06T02:38:41.776 INFO:teuthology.orchestra.run.smithi184.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-06T02:38:41.776 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-argparse x86_64 2:18.2.1-0.el9 ceph 45 k 2024-04-06T02:38:41.776 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-common x86_64 2:18.2.1-0.el9 ceph 124 k 2024-04-06T02:38:41.776 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cephfs x86_64 2:18.2.1-0.el9 ceph 161 k 2024-04-06T02:38:41.776 INFO:teuthology.orchestra.run.smithi184.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-06T02:38:41.776 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rados x86_64 2:18.2.1-0.el9 ceph 321 k 2024-04-06T02:38:41.776 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rbd x86_64 2:18.2.1-0.el9 ceph 297 k 2024-04-06T02:38:41.776 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rgw x86_64 2:18.2.1-0.el9 ceph 99 k 2024-04-06T02:38:41.776 INFO:teuthology.orchestra.run.smithi184.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-06T02:38:41.776 INFO:teuthology.orchestra.run.smithi184.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-06T02:38:41.777 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:38:41.777 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-04-06T02:38:41.777 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:38:41.777 INFO:teuthology.orchestra.run.smithi184.stdout:Install 28 Packages 2024-04-06T02:38:41.777 INFO:teuthology.orchestra.run.smithi184.stdout:Upgrade 2 Packages 2024-04-06T02:38:41.777 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:38:41.777 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 53 M 2024-04-06T02:38:41.778 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-04-06T02:38:42.108 INFO:teuthology.orchestra.run.smithi184.stdout:(1/30): ceph-base-18.2.1-0.el9.x86_64.rpm 28 MB/s | 5.2 MB 00:00 2024-04-06T02:38:42.125 INFO:teuthology.orchestra.run.smithi184.stdout:(2/30): ceph-selinux-18.2.1-0.el9.x86_64.rpm 1.4 MB/s | 24 kB 00:00 2024-04-06T02:38:42.158 INFO:teuthology.orchestra.run.smithi184.stdout:(3/30): libcephfs2-18.2.1-0.el9.x86_64.rpm 19 MB/s | 658 kB 00:00 2024-04-06T02:38:42.192 INFO:teuthology.orchestra.run.smithi184.stdout:(4/30): libradosstriper1-18.2.1-0.el9.x86_64.rp 14 MB/s | 474 kB 00:00 2024-04-06T02:38:42.301 INFO:teuthology.orchestra.run.smithi184.stdout:(5/30): ceph-radosgw-18.2.1-0.el9.x86_64.rpm 20 MB/s | 7.7 MB 00:00 2024-04-06T02:38:42.318 INFO:teuthology.orchestra.run.smithi184.stdout:(6/30): python3-ceph-argparse-18.2.1-0.el9.x86_ 2.7 MB/s | 45 kB 00:00 2024-04-06T02:38:42.343 INFO:teuthology.orchestra.run.smithi184.stdout:(7/30): python3-ceph-common-18.2.1-0.el9.x86_64 4.9 MB/s | 124 kB 00:00 2024-04-06T02:38:42.368 INFO:teuthology.orchestra.run.smithi184.stdout:(8/30): python3-cephfs-18.2.1-0.el9.x86_64.rpm 6.4 MB/s | 161 kB 00:00 2024-04-06T02:38:42.394 INFO:teuthology.orchestra.run.smithi184.stdout:(9/30): python3-rados-18.2.1-0.el9.x86_64.rpm 13 MB/s | 321 kB 00:00 2024-04-06T02:38:42.419 INFO:teuthology.orchestra.run.smithi184.stdout:(10/30): python3-rbd-18.2.1-0.el9.x86_64.rpm 12 MB/s | 297 kB 00:00 2024-04-06T02:38:42.444 INFO:teuthology.orchestra.run.smithi184.stdout:(11/30): python3-rgw-18.2.1-0.el9.x86_64.rpm 3.9 MB/s | 99 kB 00:00 2024-04-06T02:38:42.511 INFO:teuthology.orchestra.run.smithi184.stdout:(12/30): librgw2-18.2.1-0.el9.x86_64.rpm 14 MB/s | 4.5 MB 00:00 2024-04-06T02:38:42.720 INFO:teuthology.orchestra.run.smithi184.stdout:(13/30): ceph-common-18.2.1-0.el9.x86_64.rpm 23 MB/s | 18 MB 00:00 2024-04-06T02:38:42.778 INFO:teuthology.orchestra.run.smithi184.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 320 kB/s | 107 kB 00:00 2024-04-06T02:38:42.870 INFO:teuthology.orchestra.run.smithi184.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 302 kB/s | 45 kB 00:00 2024-04-06T02:38:42.895 INFO:teuthology.orchestra.run.smithi184.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 417 kB/s | 160 kB 00:00 2024-04-06T02:38:42.946 INFO:teuthology.orchestra.run.smithi184.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 3.9 MB/s | 662 kB 00:00 2024-04-06T02:38:42.963 INFO:teuthology.orchestra.run.smithi184.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 630 kB/s | 42 kB 00:00 2024-04-06T02:38:43.005 INFO:teuthology.orchestra.run.smithi184.stdout:(19/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-04-06T02:38:43.261 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-04-06T02:38:43.299 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-04-06T02:38:43.521 INFO:teuthology.orchestra.run.smithi184.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 192 kB/s | 111 kB 00:00 2024-04-06T02:38:43.638 INFO:teuthology.orchestra.run.smithi184.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 222 kB/s | 26 kB 00:00 2024-04-06T02:38:43.839 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-04-06T02:38:43.856 INFO:teuthology.orchestra.run.smithi184.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 224 kB/s | 49 kB 00:00 2024-04-06T02:38:43.886 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-04-06T02:38:43.965 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-04-06T02:38:43.965 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-06T02:38:43.965 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-06T02:38:43.965 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:38:43.981 INFO:teuthology.orchestra.run.smithi184.stdout:(23/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 303 kB/s | 309 kB 00:01 2024-04-06T02:38:44.064 INFO:teuthology.orchestra.run.smithi184.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 323 kB/s | 67 kB 00:00 2024-04-06T02:38:44.137 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-04-06T02:38:44.531 INFO:teuthology.orchestra.run.smithi184.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 411 kB/s | 192 kB 00:00 2024-04-06T02:38:45.556 INFO:teuthology.orchestra.run.smithi184.stdout:(26/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 532 kB/s | 838 kB 00:01 2024-04-06T02:38:45.632 INFO:teuthology.orchestra.run.smithi184.stdout:(27/30): librados2-18.2.1-0.el9.x86_64.rpm 44 MB/s | 3.3 MB 00:00 2024-04-06T02:38:45.690 INFO:teuthology.orchestra.run.smithi184.stdout:(28/30): librbd1-18.2.1-0.el9.x86_64.rpm 52 MB/s | 3.0 MB 00:00 2024-04-06T02:38:48.998 INFO:teuthology.orchestra.run.smithi184.stdout:(29/30): thrift-0.15.0-2.el9.x86_64.rpm 363 kB/s | 1.6 MB 00:04 2024-04-06T02:38:50.807 INFO:teuthology.orchestra.run.smithi184.stdout:(30/30): libarrow-9.0.0-10.el9.x86_64.rpm 580 kB/s | 4.4 MB 00:07 2024-04-06T02:38:50.808 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:38:50.809 INFO:teuthology.orchestra.run.smithi184.stdout:Total 5.9 MB/s | 53 MB 00:09 2024-04-06T02:38:50.877 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-04-06T02:38:50.912 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-04-06T02:38:50.912 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-04-06T02:38:51.353 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-04-06T02:38:51.353 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-04-06T02:38:52.728 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-04-06T02:38:52.806 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-04-06T02:38:52.845 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-04-06T02:38:52.903 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-04-06T02:38:53.270 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-04-06T02:38:53.302 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : librados2-2:18.2.1-0.el9.x86_64 5/32 2024-04-06T02:38:53.378 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librados2-2:18.2.1-0.el9.x86_64 5/32 2024-04-06T02:38:53.482 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rados-2:18.2.1-0.el9.x86_64 6/32 2024-04-06T02:38:53.521 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-04-06T02:38:53.567 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-04-06T02:38:53.595 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-04-06T02:38:53.644 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 9/32 2024-04-06T02:38:53.710 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-04-06T02:38:53.710 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /sys 2024-04-06T02:38:53.710 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /proc 2024-04-06T02:38:53.710 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /mnt 2024-04-06T02:38:53.711 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /var/tmp 2024-04-06T02:38:53.711 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /home 2024-04-06T02:38:53.711 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /root 2024-04-06T02:38:53.711 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /tmp 2024-04-06T02:38:53.711 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:38:53.715 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-04-06T02:38:53.740 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-04-06T02:38:53.805 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-04-06T02:38:53.911 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-04-06T02:38:53.948 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-04-06T02:38:53.987 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-04-06T02:38:54.012 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-04-06T02:38:54.047 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-04-06T02:38:54.083 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-04-06T02:38:54.116 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-04-06T02:38:54.414 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-04-06T02:38:54.440 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-04-06T02:38:54.445 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-04-06T02:38:54.464 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-04-06T02:38:54.465 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-06T02:38:54.465 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-06T02:38:54.465 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-06T02:38:54.465 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-06T02:38:54.465 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:38:54.467 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-06T02:38:54.518 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-04-06T02:38:54.550 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rbd-2:18.2.1-0.el9.x86_64 21/32 2024-04-06T02:38:54.551 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-06T02:38:54.551 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-06T02:38:54.586 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-04-06T02:38:54.986 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-04-06T02:38:55.015 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-04-06T02:38:55.086 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-04-06T02:38:55.161 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rgw-2:18.2.1-0.el9.x86_64 25/32 2024-04-06T02:38:55.184 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-ceph-common-2:18.2.1-0.el9.x86_64 26/32 2024-04-06T02:38:55.660 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-06T02:38:55.660 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-base-2:18.2.1-0.el9.x86_64 1/32 2024-04-06T02:38:55.660 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-common-2:18.2.1-0.el9.x86_64 2/32 2024-04-06T02:38:55.661 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-radosgw-2:18.2.1-0.el9.x86_64 3/32 2024-04-06T02:38:55.661 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-selinux-2:18.2.1-0.el9.x86_64 4/32 2024-04-06T02:38:55.661 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs2-2:18.2.1-0.el9.x86_64 5/32 2024-04-06T02:38:55.661 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libradosstriper1-2:18.2.1-0.el9.x86_64 6/32 2024-04-06T02:38:55.661 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librgw2-2:18.2.1-0.el9.x86_64 7/32 2024-04-06T02:38:55.661 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 8/32 2024-04-06T02:38:55.661 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-common-2:18.2.1-0.el9.x86_64 9/32 2024-04-06T02:38:55.661 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-04-06T02:38:55.661 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rados-2:18.2.1-0.el9.x86_64 11/32 2024-04-06T02:38:55.661 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rbd-2:18.2.1-0.el9.x86_64 12/32 2024-04-06T02:38:55.661 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rgw-2:18.2.1-0.el9.x86_64 13/32 2024-04-06T02:38:55.661 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-04-06T02:38:55.661 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-04-06T02:38:55.661 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-04-06T02:38:55.662 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-04-06T02:38:55.662 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-04-06T02:38:55.662 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-04-06T02:38:55.662 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-04-06T02:38:55.662 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-04-06T02:38:55.662 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-04-06T02:38:55.662 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-04-06T02:38:55.662 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-04-06T02:38:55.662 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-04-06T02:38:55.662 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-04-06T02:38:55.662 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-04-06T02:38:55.663 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-04-06T02:38:55.663 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-2:18.2.1-0.el9.x86_64 29/32 2024-04-06T02:38:55.663 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-04-06T02:38:55.663 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-2:18.2.1-0.el9.x86_64 31/32 2024-04-06T02:38:56.724 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-04-06T02:38:56.724 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:38:56.724 INFO:teuthology.orchestra.run.smithi105.stdout:Upgraded: 2024-04-06T02:38:56.724 INFO:teuthology.orchestra.run.smithi105.stdout: librados2-2:18.2.1-0.el9.x86_64 librbd1-2:18.2.1-0.el9.x86_64 2024-04-06T02:38:56.725 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-06T02:38:56.725 INFO:teuthology.orchestra.run.smithi105.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-06T02:38:56.725 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base-2:18.2.1-0.el9.x86_64 2024-04-06T02:38:56.725 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common-2:18.2.1-0.el9.x86_64 2024-04-06T02:38:56.725 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw-2:18.2.1-0.el9.x86_64 2024-04-06T02:38:56.725 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux-2:18.2.1-0.el9.x86_64 2024-04-06T02:38:56.725 INFO:teuthology.orchestra.run.smithi105.stdout: fmt-8.1.1-5.el9.x86_64 2024-04-06T02:38:56.725 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-06T02:38:56.725 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-06T02:38:56.725 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-06T02:38:56.725 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2-2:18.2.1-0.el9.x86_64 2024-04-06T02:38:56.725 INFO:teuthology.orchestra.run.smithi105.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-06T02:38:56.725 INFO:teuthology.orchestra.run.smithi105.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-06T02:38:56.725 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-06T02:38:56.726 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1-2:18.2.1-0.el9.x86_64 2024-04-06T02:38:56.726 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-06T02:38:56.726 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2-2:18.2.1-0.el9.x86_64 2024-04-06T02:38:56.726 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-06T02:38:56.726 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-06T02:38:56.726 INFO:teuthology.orchestra.run.smithi105.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-06T02:38:56.726 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse-2:18.2.1-0.el9.x86_64 2024-04-06T02:38:56.726 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common-2:18.2.1-0.el9.x86_64 2024-04-06T02:38:56.726 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs-2:18.2.1-0.el9.x86_64 2024-04-06T02:38:56.726 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-06T02:38:56.726 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados-2:18.2.1-0.el9.x86_64 2024-04-06T02:38:56.726 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd-2:18.2.1-0.el9.x86_64 2024-04-06T02:38:56.726 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw-2:18.2.1-0.el9.x86_64 2024-04-06T02:38:56.726 INFO:teuthology.orchestra.run.smithi105.stdout: re2-1:20211101-3.el9.x86_64 2024-04-06T02:38:56.727 INFO:teuthology.orchestra.run.smithi105.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-06T02:38:56.727 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:38:56.727 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-06T02:38:57.064 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-test 2024-04-06T02:38:57.263 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-04-06T02:38:57.306 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-04-06T02:38:57.654 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:31 ago on Sat 06 Apr 2024 02:38:26 AM UTC. 2024-04-06T02:38:57.850 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-04-06T02:38:57.892 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-04-06T02:38:57.943 INFO:teuthology.orchestra.run.smithi105.stderr:Modular dependency problems: 2024-04-06T02:38:57.943 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-06T02:38:57.943 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 1: conflicting requests 2024-04-06T02:38:57.943 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.943 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 2: conflicting requests 2024-04-06T02:38:57.943 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.943 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 3: conflicting requests 2024-04-06T02:38:57.943 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.943 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 4: conflicting requests 2024-04-06T02:38:57.944 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.944 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 5: conflicting requests 2024-04-06T02:38:57.944 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.944 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 6: conflicting requests 2024-04-06T02:38:57.944 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.944 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 7: conflicting requests 2024-04-06T02:38:57.944 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.944 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 8: conflicting requests 2024-04-06T02:38:57.944 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.944 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 9: conflicting requests 2024-04-06T02:38:57.944 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.944 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 10: conflicting requests 2024-04-06T02:38:57.944 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.945 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 11: conflicting requests 2024-04-06T02:38:57.945 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.945 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 12: conflicting requests 2024-04-06T02:38:57.945 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.945 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 13: conflicting requests 2024-04-06T02:38:57.945 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.945 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 14: conflicting requests 2024-04-06T02:38:57.945 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.945 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 15: conflicting requests 2024-04-06T02:38:57.945 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.945 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 16: conflicting requests 2024-04-06T02:38:57.945 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.945 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 17: conflicting requests 2024-04-06T02:38:57.946 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.946 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 18: conflicting requests 2024-04-06T02:38:57.946 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.946 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 19: conflicting requests 2024-04-06T02:38:57.946 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.946 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 20: conflicting requests 2024-04-06T02:38:57.946 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.946 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 21: conflicting requests 2024-04-06T02:38:57.946 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.946 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 22: conflicting requests 2024-04-06T02:38:57.946 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.946 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 23: conflicting requests 2024-04-06T02:38:57.946 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.946 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 24: conflicting requests 2024-04-06T02:38:57.947 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.947 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 25: conflicting requests 2024-04-06T02:38:57.947 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.947 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 26: conflicting requests 2024-04-06T02:38:57.947 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.947 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 27: conflicting requests 2024-04-06T02:38:57.947 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.947 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 28: conflicting requests 2024-04-06T02:38:57.947 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.947 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 29: conflicting requests 2024-04-06T02:38:57.947 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.947 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 30: conflicting requests 2024-04-06T02:38:57.947 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.948 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 31: conflicting requests 2024-04-06T02:38:57.948 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.948 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 32: conflicting requests 2024-04-06T02:38:57.948 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.948 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 33: conflicting requests 2024-04-06T02:38:57.948 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.948 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 34: conflicting requests 2024-04-06T02:38:57.948 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.948 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 35: conflicting requests 2024-04-06T02:38:57.948 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.948 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 36: conflicting requests 2024-04-06T02:38:57.948 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.948 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 37: conflicting requests 2024-04-06T02:38:57.949 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.949 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.949 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.949 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:38:57.949 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.949 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.949 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.949 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.949 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 39: conflicting requests 2024-04-06T02:38:57.949 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.949 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.949 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.949 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.949 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.950 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.950 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:38:57.950 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.950 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.950 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.950 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.950 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:38:57.950 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.950 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.950 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.950 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.950 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:38:57.950 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.951 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.951 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.951 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.951 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:38:57.951 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.951 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.951 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.951 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.951 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:38:57.951 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.951 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.951 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.951 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.952 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:38:57.952 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.952 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.952 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.952 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:38:57.971 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-04-06T02:38:57.971 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-06T02:38:57.971 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-06T02:38:57.971 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:38:58.010 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-04-06T02:38:58.011 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-06T02:38:58.012 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:38:58.012 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2024-04-06T02:38:58.012 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:38:58.012 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-06T02:38:58.012 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test x86_64 2:18.2.1-0.el9 ceph 40 M 2024-04-06T02:38:58.012 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-04-06T02:38:58.012 INFO:teuthology.orchestra.run.smithi105.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-06T02:38:58.013 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite x86_64 2:18.2.1-0.el9 ceph 165 k 2024-04-06T02:38:58.013 INFO:teuthology.orchestra.run.smithi105.stdout: oniguruma x86_64 6.9.6-1.el9.5 baseos 218 k 2024-04-06T02:38:58.013 INFO:teuthology.orchestra.run.smithi105.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-06T02:38:58.013 INFO:teuthology.orchestra.run.smithi105.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-06T02:38:58.013 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:38:58.013 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-06T02:38:58.013 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:38:58.013 INFO:teuthology.orchestra.run.smithi105.stdout:Install 6 Packages 2024-04-06T02:38:58.013 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:38:58.014 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 41 M 2024-04-06T02:38:58.014 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 167 M 2024-04-06T02:38:58.014 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-06T02:38:58.375 INFO:teuthology.orchestra.run.smithi105.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 948 kB/s | 187 kB 00:00 2024-04-06T02:38:58.433 INFO:teuthology.orchestra.run.smithi105.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.7 MB/s | 218 kB 00:00 2024-04-06T02:38:58.459 INFO:teuthology.orchestra.run.smithi105.stdout:(3/6): libcephsqlite-18.2.1-0.el9.x86_64.rpm 586 kB/s | 165 kB 00:00 2024-04-06T02:38:58.559 INFO:teuthology.orchestra.run.smithi105.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 634 kB/s | 64 kB 00:00 2024-04-06T02:38:58.601 INFO:teuthology.orchestra.run.smithi105.stdout:(5/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.8 MB/s | 305 kB 00:00 2024-04-06T02:38:59.251 INFO:teuthology.orchestra.run.smithi105.stdout:(6/6): ceph-test-18.2.1-0.el9.x86_64.rpm 37 MB/s | 40 MB 00:01 2024-04-06T02:38:59.254 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:38:59.254 INFO:teuthology.orchestra.run.smithi105.stdout:Total 33 MB/s | 41 MB 00:01 2024-04-06T02:38:59.304 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-06T02:38:59.323 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-06T02:38:59.324 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-06T02:38:59.671 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-06T02:38:59.672 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-06T02:39:00.238 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-06T02:39:00.307 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-04-06T02:39:00.383 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-04-06T02:39:00.418 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-06T02:39:00.461 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-04-06T02:39:00.488 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephsqlite-2:18.2.1-0.el9.x86_64 5/6 2024-04-06T02:39:04.251 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephsqlite-2:18.2.1-0.el9.x86_64 5/6 2024-04-06T02:39:04.339 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-test-2:18.2.1-0.el9.x86_64 6/6 2024-04-06T02:39:06.567 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-test-2:18.2.1-0.el9.x86_64 6/6 2024-04-06T02:39:06.567 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-test-2:18.2.1-0.el9.x86_64 1/6 2024-04-06T02:39:06.567 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephsqlite-2:18.2.1-0.el9.x86_64 2/6 2024-04-06T02:39:06.567 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2024-04-06T02:39:06.567 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2024-04-06T02:39:06.567 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2024-04-06T02:39:06.966 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2024-04-06T02:39:06.966 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:06.966 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-06T02:39:06.966 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test-2:18.2.1-0.el9.x86_64 jq-1.6-16.el9.x86_64 2024-04-06T02:39:06.966 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite-2:18.2.1-0.el9.x86_64 oniguruma-6.9.6-1.el9.5.x86_64 2024-04-06T02:39:06.966 INFO:teuthology.orchestra.run.smithi105.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-04-06T02:39:06.966 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:06.967 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-06T02:39:07.220 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph 2024-04-06T02:39:07.369 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-04-06T02:39:07.370 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /sys 2024-04-06T02:39:07.370 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /proc 2024-04-06T02:39:07.370 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /mnt 2024-04-06T02:39:07.370 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /var/tmp 2024-04-06T02:39:07.370 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /home 2024-04-06T02:39:07.370 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /root 2024-04-06T02:39:07.370 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /tmp 2024-04-06T02:39:07.370 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:07.816 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:41 ago on Sat 06 Apr 2024 02:38:26 AM UTC. 2024-04-06T02:39:08.104 INFO:teuthology.orchestra.run.smithi105.stderr:Modular dependency problems: 2024-04-06T02:39:08.105 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-06T02:39:08.105 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 1: conflicting requests 2024-04-06T02:39:08.105 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.105 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 2: conflicting requests 2024-04-06T02:39:08.105 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.105 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 3: conflicting requests 2024-04-06T02:39:08.105 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.105 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 4: conflicting requests 2024-04-06T02:39:08.105 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.105 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 5: conflicting requests 2024-04-06T02:39:08.105 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.105 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 6: conflicting requests 2024-04-06T02:39:08.106 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.106 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 7: conflicting requests 2024-04-06T02:39:08.106 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.106 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 8: conflicting requests 2024-04-06T02:39:08.106 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.106 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 9: conflicting requests 2024-04-06T02:39:08.106 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.106 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 10: conflicting requests 2024-04-06T02:39:08.106 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.106 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 11: conflicting requests 2024-04-06T02:39:08.106 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.106 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 12: conflicting requests 2024-04-06T02:39:08.106 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.107 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 13: conflicting requests 2024-04-06T02:39:08.107 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.107 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 14: conflicting requests 2024-04-06T02:39:08.107 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.107 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 15: conflicting requests 2024-04-06T02:39:08.107 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.107 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 16: conflicting requests 2024-04-06T02:39:08.107 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.107 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 17: conflicting requests 2024-04-06T02:39:08.107 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.107 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 18: conflicting requests 2024-04-06T02:39:08.107 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.108 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 19: conflicting requests 2024-04-06T02:39:08.108 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.108 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 20: conflicting requests 2024-04-06T02:39:08.108 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.108 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 21: conflicting requests 2024-04-06T02:39:08.108 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.108 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 22: conflicting requests 2024-04-06T02:39:08.108 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.108 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 23: conflicting requests 2024-04-06T02:39:08.108 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.108 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 24: conflicting requests 2024-04-06T02:39:08.108 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.108 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 25: conflicting requests 2024-04-06T02:39:08.109 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.109 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 26: conflicting requests 2024-04-06T02:39:08.109 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.109 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 27: conflicting requests 2024-04-06T02:39:08.109 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.109 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 28: conflicting requests 2024-04-06T02:39:08.109 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.109 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 29: conflicting requests 2024-04-06T02:39:08.109 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.109 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 30: conflicting requests 2024-04-06T02:39:08.109 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.109 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 31: conflicting requests 2024-04-06T02:39:08.109 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.109 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 32: conflicting requests 2024-04-06T02:39:08.110 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.110 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 33: conflicting requests 2024-04-06T02:39:08.110 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.110 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 34: conflicting requests 2024-04-06T02:39:08.110 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.110 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 35: conflicting requests 2024-04-06T02:39:08.110 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.111 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 36: conflicting requests 2024-04-06T02:39:08.111 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.111 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 37: conflicting requests 2024-04-06T02:39:08.111 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.111 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.111 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.111 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:39:08.111 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.111 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.111 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.111 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.111 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 39: conflicting requests 2024-04-06T02:39:08.111 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.112 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.112 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.112 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.112 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.112 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.112 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:39:08.112 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.112 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.112 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.112 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.112 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:39:08.112 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.112 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.113 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.113 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.113 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:39:08.113 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.113 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.113 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.113 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.113 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:39:08.113 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.113 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.113 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.113 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.113 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:39:08.114 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.114 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.114 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.114 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.114 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:39:08.114 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.114 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.114 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.114 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:39:08.119 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-04-06T02:39:08.144 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-04-06T02:39:08.144 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-06T02:39:08.144 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-06T02:39:08.144 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-06T02:39:08.145 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-06T02:39:08.145 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:08.146 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-06T02:39:08.178 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-06T02:39:08.180 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:39:08.180 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-04-06T02:39:08.180 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:39:08.180 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-06T02:39:08.181 INFO:teuthology.orchestra.run.smithi105.stdout: ceph x86_64 2:18.2.1-0.el9 ceph 6.4 k 2024-04-06T02:39:08.181 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-04-06T02:39:08.181 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds x86_64 2:18.2.1-0.el9 ceph 2.1 M 2024-04-06T02:39:08.181 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr x86_64 2:18.2.1-0.el9 ceph 1.4 M 2024-04-06T02:39:08.181 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core noarch 2:18.2.1-0.el9 ceph-noarch 242 k 2024-04-06T02:39:08.181 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon x86_64 2:18.2.1-0.el9 ceph 4.4 M 2024-04-06T02:39:08.181 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd x86_64 2:18.2.1-0.el9 ceph 18 M 2024-04-06T02:39:08.181 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-06T02:39:08.181 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-06T02:39:08.181 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-06T02:39:08.181 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-06T02:39:08.181 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-06T02:39:08.182 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-06T02:39:08.182 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-06T02:39:08.182 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-06T02:39:08.182 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-06T02:39:08.182 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-06T02:39:08.182 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-06T02:39:08.182 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-06T02:39:08.182 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-06T02:39:08.182 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-06T02:39:08.182 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-06T02:39:08.182 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-06T02:39:08.183 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-06T02:39:08.183 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-06T02:39:08.183 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-06T02:39:08.183 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-06T02:39:08.183 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-06T02:39:08.183 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-06T02:39:08.183 INFO:teuthology.orchestra.run.smithi105.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-06T02:39:08.183 INFO:teuthology.orchestra.run.smithi105.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-06T02:39:08.183 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-06T02:39:08.183 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-06T02:39:08.183 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-06T02:39:08.183 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:08.184 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-06T02:39:08.184 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:39:08.184 INFO:teuthology.orchestra.run.smithi105.stdout:Install 33 Packages 2024-04-06T02:39:08.184 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:08.184 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 30 M 2024-04-06T02:39:08.184 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 110 M 2024-04-06T02:39:08.185 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-06T02:39:08.231 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-06T02:39:08.231 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-06T02:39:08.639 INFO:teuthology.orchestra.run.smithi105.stdout:(1/33): ceph-18.2.1-0.el9.x86_64.rpm 49 kB/s | 6.4 kB 00:00 2024-04-06T02:39:08.731 INFO:teuthology.orchestra.run.smithi105.stdout:(2/33): ceph-mds-18.2.1-0.el9.x86_64.rpm 9.4 MB/s | 2.1 MB 00:00 2024-04-06T02:39:08.865 INFO:teuthology.orchestra.run.smithi105.stdout:(3/33): ceph-mon-18.2.1-0.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-04-06T02:39:09.057 INFO:teuthology.orchestra.run.smithi105.stdout:(4/33): ceph-mgr-18.2.1-0.el9.x86_64.rpm 2.6 MB/s | 1.4 MB 00:00 2024-04-06T02:39:09.082 INFO:teuthology.orchestra.run.smithi105.stdout:(5/33): ceph-mgr-modules-core-18.2.1-0.el9.noar 1.1 MB/s | 242 kB 00:00 2024-04-06T02:39:09.291 INFO:teuthology.orchestra.run.smithi105.stdout:(6/33): ceph-osd-18.2.1-0.el9.x86_64.rpm 31 MB/s | 18 MB 00:00 2024-04-06T02:39:09.340 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-06T02:39:09.340 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-base-2:18.2.1-0.el9.x86_64 1/32 2024-04-06T02:39:09.340 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-common-2:18.2.1-0.el9.x86_64 2/32 2024-04-06T02:39:09.340 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-radosgw-2:18.2.1-0.el9.x86_64 3/32 2024-04-06T02:39:09.340 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-selinux-2:18.2.1-0.el9.x86_64 4/32 2024-04-06T02:39:09.340 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephfs2-2:18.2.1-0.el9.x86_64 5/32 2024-04-06T02:39:09.340 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libradosstriper1-2:18.2.1-0.el9.x86_64 6/32 2024-04-06T02:39:09.340 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librgw2-2:18.2.1-0.el9.x86_64 7/32 2024-04-06T02:39:09.340 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 8/32 2024-04-06T02:39:09.340 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-ceph-common-2:18.2.1-0.el9.x86_64 9/32 2024-04-06T02:39:09.340 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-04-06T02:39:09.341 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rados-2:18.2.1-0.el9.x86_64 11/32 2024-04-06T02:39:09.341 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rbd-2:18.2.1-0.el9.x86_64 12/32 2024-04-06T02:39:09.341 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rgw-2:18.2.1-0.el9.x86_64 13/32 2024-04-06T02:39:09.341 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-04-06T02:39:09.341 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-04-06T02:39:09.341 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-04-06T02:39:09.341 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-04-06T02:39:09.341 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-04-06T02:39:09.341 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-04-06T02:39:09.341 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-04-06T02:39:09.341 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-04-06T02:39:09.341 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-04-06T02:39:09.341 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-04-06T02:39:09.341 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-04-06T02:39:09.342 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-04-06T02:39:09.342 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-04-06T02:39:09.342 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-04-06T02:39:09.342 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-04-06T02:39:09.342 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados2-2:18.2.1-0.el9.x86_64 29/32 2024-04-06T02:39:09.342 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-04-06T02:39:09.342 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librbd1-2:18.2.1-0.el9.x86_64 31/32 2024-04-06T02:39:09.483 INFO:teuthology.orchestra.run.smithi105.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 595 kB/s | 253 kB 00:00 2024-04-06T02:39:09.508 INFO:teuthology.orchestra.run.smithi105.stdout:(8/33): python3-ply-3.11-14.el9.noarch.rpm 491 kB/s | 106 kB 00:00 2024-04-06T02:39:09.542 INFO:teuthology.orchestra.run.smithi105.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 2.3 MB/s | 135 kB 00:00 2024-04-06T02:39:09.600 INFO:teuthology.orchestra.run.smithi105.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 1.3 MB/s | 125 kB 00:00 2024-04-06T02:39:09.634 INFO:teuthology.orchestra.run.smithi105.stdout:(11/33): python3-urllib3-1.26.5-5.el9.noarch.rp 2.3 MB/s | 215 kB 00:00 2024-04-06T02:39:09.676 INFO:teuthology.orchestra.run.smithi105.stdout:(12/33): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-04-06T02:39:09.701 INFO:teuthology.orchestra.run.smithi105.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 520 kB/s | 35 kB 00:00 2024-04-06T02:39:09.735 INFO:teuthology.orchestra.run.smithi105.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 871 kB/s | 51 kB 00:00 2024-04-06T02:39:09.768 INFO:teuthology.orchestra.run.smithi105.stdout:(15/33): python3-cryptography-36.0.1-4.el9.x86_ 1.8 MB/s | 1.2 MB 00:00 2024-04-06T02:39:09.793 INFO:teuthology.orchestra.run.smithi105.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 452 kB/s | 42 kB 00:00 2024-04-06T02:39:10.019 INFO:teuthology.orchestra.run.smithi105.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 153 kB/s | 43 kB 00:00 2024-04-06T02:39:10.094 INFO:teuthology.orchestra.run.smithi105.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 143 kB/s | 11 kB 00:00 2024-04-06T02:39:10.161 INFO:teuthology.orchestra.run.smithi105.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 437 kB/s | 172 kB 00:00 2024-04-06T02:39:10.195 INFO:teuthology.orchestra.run.smithi105.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 176 kB/s | 18 kB 00:00 2024-04-06T02:39:10.253 INFO:teuthology.orchestra.run.smithi105.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 253 kB/s | 23 kB 00:00 2024-04-06T02:39:10.270 INFO:teuthology.orchestra.run.smithi105.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 259 kB/s | 19 kB 00:00 2024-04-06T02:39:10.354 INFO:teuthology.orchestra.run.smithi105.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-04-06T02:39:10.371 INFO:teuthology.orchestra.run.smithi105.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 463 kB/s | 46 kB 00:00 2024-04-06T02:39:10.413 INFO:teuthology.orchestra.run.smithi105.stdout:(25/33): python3-cherrypy-18.6.1-2.el9.noarch.r 579 kB/s | 358 kB 00:00 2024-04-06T02:39:10.438 INFO:teuthology.orchestra.run.smithi105.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 943 kB/s | 79 kB 00:00 2024-04-06T02:39:10.579 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-04-06T02:39:10.579 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:10.579 INFO:teuthology.orchestra.run.smithi184.stdout:Upgraded: 2024-04-06T02:39:10.579 INFO:teuthology.orchestra.run.smithi184.stdout: librados2-2:18.2.1-0.el9.x86_64 librbd1-2:18.2.1-0.el9.x86_64 2024-04-06T02:39:10.579 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-04-06T02:39:10.580 INFO:teuthology.orchestra.run.smithi184.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-06T02:39:10.580 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-base-2:18.2.1-0.el9.x86_64 2024-04-06T02:39:10.580 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-common-2:18.2.1-0.el9.x86_64 2024-04-06T02:39:10.580 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-radosgw-2:18.2.1-0.el9.x86_64 2024-04-06T02:39:10.580 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-selinux-2:18.2.1-0.el9.x86_64 2024-04-06T02:39:10.580 INFO:teuthology.orchestra.run.smithi184.stdout: fmt-8.1.1-5.el9.x86_64 2024-04-06T02:39:10.580 INFO:teuthology.orchestra.run.smithi184.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-06T02:39:10.580 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-06T02:39:10.580 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-06T02:39:10.580 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs2-2:18.2.1-0.el9.x86_64 2024-04-06T02:39:10.580 INFO:teuthology.orchestra.run.smithi184.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-06T02:39:10.580 INFO:teuthology.orchestra.run.smithi184.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-06T02:39:10.580 INFO:teuthology.orchestra.run.smithi184.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-06T02:39:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: libradosstriper1-2:18.2.1-0.el9.x86_64 2024-04-06T02:39:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-06T02:39:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: librgw2-2:18.2.1-0.el9.x86_64 2024-04-06T02:39:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-06T02:39:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-06T02:39:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-06T02:39:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-argparse-2:18.2.1-0.el9.x86_64 2024-04-06T02:39:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-common-2:18.2.1-0.el9.x86_64 2024-04-06T02:39:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cephfs-2:18.2.1-0.el9.x86_64 2024-04-06T02:39:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-06T02:39:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rados-2:18.2.1-0.el9.x86_64 2024-04-06T02:39:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rbd-2:18.2.1-0.el9.x86_64 2024-04-06T02:39:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rgw-2:18.2.1-0.el9.x86_64 2024-04-06T02:39:10.581 INFO:teuthology.orchestra.run.smithi184.stdout: re2-1:20211101-3.el9.x86_64 2024-04-06T02:39:10.582 INFO:teuthology.orchestra.run.smithi184.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-06T02:39:10.582 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:10.582 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-04-06T02:39:10.596 INFO:teuthology.orchestra.run.smithi105.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.2 MB/s | 272 kB 00:00 2024-04-06T02:39:10.647 INFO:teuthology.orchestra.run.smithi105.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 70 kB/s | 16 kB 00:00 2024-04-06T02:39:10.672 INFO:teuthology.orchestra.run.smithi105.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 385 kB/s | 90 kB 00:00 2024-04-06T02:39:10.706 INFO:teuthology.orchestra.run.smithi105.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 329 kB/s | 36 kB 00:00 2024-04-06T02:39:10.848 INFO:teuthology.orchestra.run.smithi105.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.1 MB/s | 230 kB 00:00 2024-04-06T02:39:10.920 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-test 2024-04-06T02:39:10.981 INFO:teuthology.orchestra.run.smithi105.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.4 MB/s | 427 kB 00:00 2024-04-06T02:39:10.998 INFO:teuthology.orchestra.run.smithi105.stdout:(33/33): python3-zc-lockfile-2.0-10.el9.noarch. 68 kB/s | 20 kB 00:00 2024-04-06T02:39:11.003 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:39:11.004 INFO:teuthology.orchestra.run.smithi105.stdout:Total 11 MB/s | 30 MB 00:02 2024-04-06T02:39:11.124 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-06T02:39:11.161 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-06T02:39:11.161 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-06T02:39:11.526 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-06T02:39:11.526 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-06T02:39:11.527 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:33 ago on Sat 06 Apr 2024 02:38:38 AM UTC. 2024-04-06T02:39:11.815 INFO:teuthology.orchestra.run.smithi184.stderr:Modular dependency problems: 2024-04-06T02:39:11.816 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-04-06T02:39:11.816 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 1: conflicting requests 2024-04-06T02:39:11.816 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.816 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 2: conflicting requests 2024-04-06T02:39:11.816 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.816 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 3: conflicting requests 2024-04-06T02:39:11.816 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.816 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 4: conflicting requests 2024-04-06T02:39:11.816 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.816 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 5: conflicting requests 2024-04-06T02:39:11.816 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.816 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 6: conflicting requests 2024-04-06T02:39:11.816 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.817 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 7: conflicting requests 2024-04-06T02:39:11.817 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.817 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 8: conflicting requests 2024-04-06T02:39:11.817 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.817 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 9: conflicting requests 2024-04-06T02:39:11.817 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.817 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 10: conflicting requests 2024-04-06T02:39:11.817 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.817 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 11: conflicting requests 2024-04-06T02:39:11.817 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.817 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 12: conflicting requests 2024-04-06T02:39:11.817 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.817 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 13: conflicting requests 2024-04-06T02:39:11.817 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.818 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 14: conflicting requests 2024-04-06T02:39:11.818 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.818 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 15: conflicting requests 2024-04-06T02:39:11.818 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.818 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 16: conflicting requests 2024-04-06T02:39:11.818 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.818 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 17: conflicting requests 2024-04-06T02:39:11.818 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.818 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 18: conflicting requests 2024-04-06T02:39:11.818 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.818 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 19: conflicting requests 2024-04-06T02:39:11.818 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.818 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 20: conflicting requests 2024-04-06T02:39:11.819 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.819 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 21: conflicting requests 2024-04-06T02:39:11.819 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.819 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 22: conflicting requests 2024-04-06T02:39:11.819 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.819 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 23: conflicting requests 2024-04-06T02:39:11.819 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.819 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 24: conflicting requests 2024-04-06T02:39:11.819 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.819 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 25: conflicting requests 2024-04-06T02:39:11.819 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.819 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 26: conflicting requests 2024-04-06T02:39:11.819 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.819 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 27: conflicting requests 2024-04-06T02:39:11.820 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.820 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 28: conflicting requests 2024-04-06T02:39:11.820 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.820 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 29: conflicting requests 2024-04-06T02:39:11.820 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.820 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 30: conflicting requests 2024-04-06T02:39:11.820 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.820 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 31: conflicting requests 2024-04-06T02:39:11.820 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.820 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 32: conflicting requests 2024-04-06T02:39:11.820 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.820 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 33: conflicting requests 2024-04-06T02:39:11.820 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.820 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 34: conflicting requests 2024-04-06T02:39:11.821 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.821 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 35: conflicting requests 2024-04-06T02:39:11.821 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.821 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 36: conflicting requests 2024-04-06T02:39:11.821 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.821 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 37: conflicting requests 2024-04-06T02:39:11.821 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.821 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.821 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.821 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:39:11.821 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.821 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.821 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.821 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.822 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 39: conflicting requests 2024-04-06T02:39:11.822 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.822 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.822 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.822 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.822 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.822 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.822 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:39:11.822 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.822 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.822 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.822 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.822 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:39:11.823 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.823 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.823 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.823 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.823 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:39:11.823 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.823 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.823 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.823 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.823 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:39:11.823 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.823 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.823 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.823 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.824 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:39:11.824 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.824 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.824 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.824 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.824 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:39:11.824 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.824 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.824 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.824 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:39:11.885 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-04-06T02:39:11.886 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:39:11.886 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-04-06T02:39:11.886 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:39:11.886 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-04-06T02:39:11.886 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-test x86_64 2:18.2.1-0.el9 ceph 40 M 2024-04-06T02:39:11.886 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-04-06T02:39:11.886 INFO:teuthology.orchestra.run.smithi184.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-06T02:39:11.887 INFO:teuthology.orchestra.run.smithi184.stdout: libcephsqlite x86_64 2:18.2.1-0.el9 ceph 165 k 2024-04-06T02:39:11.887 INFO:teuthology.orchestra.run.smithi184.stdout: oniguruma x86_64 6.9.6-1.el9.5 baseos 218 k 2024-04-06T02:39:11.887 INFO:teuthology.orchestra.run.smithi184.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-06T02:39:11.887 INFO:teuthology.orchestra.run.smithi184.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-06T02:39:11.887 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:11.887 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-04-06T02:39:11.887 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:39:11.887 INFO:teuthology.orchestra.run.smithi184.stdout:Install 6 Packages 2024-04-06T02:39:11.887 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:11.888 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 41 M 2024-04-06T02:39:11.888 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 167 M 2024-04-06T02:39:11.888 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-04-06T02:39:12.290 INFO:teuthology.orchestra.run.smithi184.stdout:(1/6): libcephsqlite-18.2.1-0.el9.x86_64.rpm 1.8 MB/s | 165 kB 00:00 2024-04-06T02:39:12.373 INFO:teuthology.orchestra.run.smithi184.stdout:(2/6): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-04-06T02:39:12.432 INFO:teuthology.orchestra.run.smithi184.stdout:(3/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.5 MB/s | 218 kB 00:00 2024-04-06T02:39:12.752 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-06T02:39:12.817 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-06T02:39:12.849 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-06T02:39:12.866 INFO:teuthology.orchestra.run.smithi184.stdout:(4/6): ceph-test-18.2.1-0.el9.x86_64.rpm 60 MB/s | 40 MB 00:00 2024-04-06T02:39:12.882 INFO:teuthology.orchestra.run.smithi184.stdout:(5/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 141 kB/s | 64 kB 00:00 2024-04-06T02:39:12.883 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-06T02:39:12.974 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-06T02:39:13.041 INFO:teuthology.orchestra.run.smithi184.stdout:(6/6): socat-1.7.4.1-5.el9.x86_64.rpm 457 kB/s | 305 kB 00:00 2024-04-06T02:39:13.044 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:39:13.044 INFO:teuthology.orchestra.run.smithi184.stdout:Total 35 MB/s | 41 MB 00:01 2024-04-06T02:39:13.061 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-06T02:39:13.093 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-04-06T02:39:13.101 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-06T02:39:13.113 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-04-06T02:39:13.113 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-04-06T02:39:13.145 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-06T02:39:13.175 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-04-06T02:39:13.210 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-04-06T02:39:13.243 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-04-06T02:39:13.283 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-04-06T02:39:13.319 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-04-06T02:39:13.353 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-04-06T02:39:13.398 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-04-06T02:39:13.455 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-04-06T02:39:13.455 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-04-06T02:39:13.472 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-04-06T02:39:13.618 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-04-06T02:39:13.699 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-04-06T02:39:13.753 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-04-06T02:39:13.791 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-04-06T02:39:13.840 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-04-06T02:39:13.901 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-04-06T02:39:13.997 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-04-06T02:39:14.077 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-04-06T02:39:14.097 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-04-06T02:39:14.121 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-04-06T02:39:14.153 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-04-06T02:39:14.156 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-06T02:39:14.200 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-04-06T02:39:14.223 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-04-06T02:39:14.227 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephsqlite-2:18.2.1-0.el9.x86_64 5/6 2024-04-06T02:39:14.312 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-04-06T02:39:14.358 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-04-06T02:39:14.476 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-04-06T02:39:14.637 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 28/33 2024-04-06T02:39:14.663 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-04-06T02:39:14.688 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-04-06T02:39:14.688 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-06T02:39:14.688 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-06T02:39:14.688 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-06T02:39:14.688 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-06T02:39:14.688 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:16.248 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-04-06T02:39:16.278 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-04-06T02:39:16.278 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-06T02:39:16.278 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-06T02:39:16.278 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-06T02:39:16.278 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-06T02:39:16.278 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:16.732 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-04-06T02:39:16.756 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-04-06T02:39:16.756 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-06T02:39:16.756 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-06T02:39:16.756 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-06T02:39:16.756 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-06T02:39:16.756 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:16.958 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-04-06T02:39:16.981 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-04-06T02:39:16.981 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-06T02:39:16.981 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-06T02:39:16.981 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-06T02:39:16.982 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-06T02:39:16.982 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:17.046 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-2:18.2.1-0.el9.x86_64 33/33 2024-04-06T02:39:17.967 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libcephsqlite-2:18.2.1-0.el9.x86_64 5/6 2024-04-06T02:39:18.043 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-test-2:18.2.1-0.el9.x86_64 6/6 2024-04-06T02:39:18.798 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-2:18.2.1-0.el9.x86_64 33/33 2024-04-06T02:39:18.798 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-2:18.2.1-0.el9.x86_64 1/33 2024-04-06T02:39:18.798 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mds-2:18.2.1-0.el9.x86_64 2/33 2024-04-06T02:39:18.798 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-2:18.2.1-0.el9.x86_64 3/33 2024-04-06T02:39:18.798 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mon-2:18.2.1-0.el9.x86_64 4/33 2024-04-06T02:39:18.798 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-osd-2:18.2.1-0.el9.x86_64 5/33 2024-04-06T02:39:18.798 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 6/33 2024-04-06T02:39:18.798 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-04-06T02:39:18.798 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-04-06T02:39:18.798 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-04-06T02:39:18.798 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-04-06T02:39:18.799 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-04-06T02:39:18.799 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-04-06T02:39:18.799 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-04-06T02:39:18.799 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-04-06T02:39:18.799 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-04-06T02:39:18.799 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-06T02:39:18.799 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-06T02:39:18.799 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-06T02:39:18.799 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-06T02:39:18.799 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-06T02:39:18.799 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-06T02:39:18.799 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-06T02:39:18.799 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-06T02:39:18.800 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-06T02:39:18.800 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-06T02:39:18.800 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-06T02:39:18.801 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-06T02:39:18.801 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-06T02:39:18.801 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-06T02:39:18.801 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-06T02:39:18.801 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-06T02:39:18.801 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-06T02:39:19.906 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-06T02:39:19.906 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:19.906 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-06T02:39:19.906 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-2:18.2.1-0.el9.x86_64 2024-04-06T02:39:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds-2:18.2.1-0.el9.x86_64 2024-04-06T02:39:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-2:18.2.1-0.el9.x86_64 2024-04-06T02:39:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 2024-04-06T02:39:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon-2:18.2.1-0.el9.x86_64 2024-04-06T02:39:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd-2:18.2.1-0.el9.x86_64 2024-04-06T02:39:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-06T02:39:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-06T02:39:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-06T02:39:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-06T02:39:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-06T02:39:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-06T02:39:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-06T02:39:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-06T02:39:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-06T02:39:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-06T02:39:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-06T02:39:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-06T02:39:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-06T02:39:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-06T02:39:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-06T02:39:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ply-3.11-14.el9.noarch 2024-04-06T02:39:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-06T02:39:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-06T02:39:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-06T02:39:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-06T02:39:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-06T02:39:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-06T02:39:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-06T02:39:19.909 INFO:teuthology.orchestra.run.smithi105.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-06T02:39:19.909 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-06T02:39:19.909 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-06T02:39:19.909 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-06T02:39:19.909 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:19.909 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-06T02:39:20.096 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-test-2:18.2.1-0.el9.x86_64 6/6 2024-04-06T02:39:20.096 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-test-2:18.2.1-0.el9.x86_64 1/6 2024-04-06T02:39:20.096 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephsqlite-2:18.2.1-0.el9.x86_64 2/6 2024-04-06T02:39:20.096 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2024-04-06T02:39:20.096 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2024-04-06T02:39:20.096 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2024-04-06T02:39:20.220 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-base 2024-04-06T02:39:20.525 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2024-04-06T02:39:20.525 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:20.525 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-04-06T02:39:20.525 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-test-2:18.2.1-0.el9.x86_64 jq-1.6-16.el9.x86_64 2024-04-06T02:39:20.525 INFO:teuthology.orchestra.run.smithi184.stdout: libcephsqlite-2:18.2.1-0.el9.x86_64 oniguruma-6.9.6-1.el9.5.x86_64 2024-04-06T02:39:20.526 INFO:teuthology.orchestra.run.smithi184.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-04-06T02:39:20.526 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:20.526 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-04-06T02:39:20.814 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:54 ago on Sat 06 Apr 2024 02:38:26 AM UTC. 2024-04-06T02:39:20.819 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph 2024-04-06T02:39:21.128 INFO:teuthology.orchestra.run.smithi105.stderr:Modular dependency problems: 2024-04-06T02:39:21.128 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-06T02:39:21.128 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 1: conflicting requests 2024-04-06T02:39:21.128 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.128 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 2: conflicting requests 2024-04-06T02:39:21.128 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.128 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 3: conflicting requests 2024-04-06T02:39:21.128 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.128 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 4: conflicting requests 2024-04-06T02:39:21.129 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.129 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 5: conflicting requests 2024-04-06T02:39:21.129 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.129 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 6: conflicting requests 2024-04-06T02:39:21.129 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.129 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 7: conflicting requests 2024-04-06T02:39:21.129 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.129 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 8: conflicting requests 2024-04-06T02:39:21.129 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.129 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 9: conflicting requests 2024-04-06T02:39:21.129 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.129 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 10: conflicting requests 2024-04-06T02:39:21.129 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.129 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 11: conflicting requests 2024-04-06T02:39:21.130 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.130 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 12: conflicting requests 2024-04-06T02:39:21.130 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.130 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 13: conflicting requests 2024-04-06T02:39:21.130 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.130 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 14: conflicting requests 2024-04-06T02:39:21.130 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.130 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 15: conflicting requests 2024-04-06T02:39:21.130 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.130 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 16: conflicting requests 2024-04-06T02:39:21.130 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.130 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 17: conflicting requests 2024-04-06T02:39:21.130 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.131 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 18: conflicting requests 2024-04-06T02:39:21.131 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.131 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 19: conflicting requests 2024-04-06T02:39:21.131 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.131 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 20: conflicting requests 2024-04-06T02:39:21.131 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.131 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 21: conflicting requests 2024-04-06T02:39:21.131 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.131 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 22: conflicting requests 2024-04-06T02:39:21.131 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.131 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 23: conflicting requests 2024-04-06T02:39:21.131 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.132 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 24: conflicting requests 2024-04-06T02:39:21.132 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.132 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 25: conflicting requests 2024-04-06T02:39:21.132 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.132 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 26: conflicting requests 2024-04-06T02:39:21.132 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.132 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 27: conflicting requests 2024-04-06T02:39:21.132 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.132 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 28: conflicting requests 2024-04-06T02:39:21.132 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.132 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 29: conflicting requests 2024-04-06T02:39:21.132 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.133 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 30: conflicting requests 2024-04-06T02:39:21.133 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.133 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 31: conflicting requests 2024-04-06T02:39:21.133 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.133 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 32: conflicting requests 2024-04-06T02:39:21.133 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.133 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 33: conflicting requests 2024-04-06T02:39:21.133 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.133 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 34: conflicting requests 2024-04-06T02:39:21.133 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.133 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 35: conflicting requests 2024-04-06T02:39:21.133 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.133 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 36: conflicting requests 2024-04-06T02:39:21.134 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.134 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 37: conflicting requests 2024-04-06T02:39:21.134 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.134 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.134 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.134 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:39:21.134 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.134 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.134 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.134 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.134 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 39: conflicting requests 2024-04-06T02:39:21.134 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.134 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.135 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.135 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.135 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.135 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.135 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:39:21.135 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.135 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.135 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.135 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.135 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:39:21.135 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.135 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.135 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.135 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.136 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:39:21.136 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.136 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.136 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.136 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.136 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:39:21.136 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.136 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.136 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.136 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.136 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:39:21.136 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.137 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.137 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.137 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.137 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:39:21.137 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.137 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.137 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.137 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.137 INFO:teuthology.orchestra.run.smithi105.stdout:Package ceph-base-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-06T02:39:21.195 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-06T02:39:21.196 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-04-06T02:39:21.197 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-06T02:39:21.270 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install cephadm 2024-04-06T02:39:21.419 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:43 ago on Sat 06 Apr 2024 02:38:38 AM UTC. 2024-04-06T02:39:21.712 INFO:teuthology.orchestra.run.smithi184.stderr:Modular dependency problems: 2024-04-06T02:39:21.712 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-04-06T02:39:21.712 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 1: conflicting requests 2024-04-06T02:39:21.712 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.713 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 2: conflicting requests 2024-04-06T02:39:21.713 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.713 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 3: conflicting requests 2024-04-06T02:39:21.713 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.713 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 4: conflicting requests 2024-04-06T02:39:21.713 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.713 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 5: conflicting requests 2024-04-06T02:39:21.713 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.713 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 6: conflicting requests 2024-04-06T02:39:21.713 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.713 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 7: conflicting requests 2024-04-06T02:39:21.713 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.713 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 8: conflicting requests 2024-04-06T02:39:21.713 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.714 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 9: conflicting requests 2024-04-06T02:39:21.714 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.714 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 10: conflicting requests 2024-04-06T02:39:21.714 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.714 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 11: conflicting requests 2024-04-06T02:39:21.714 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.714 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 12: conflicting requests 2024-04-06T02:39:21.714 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.714 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 13: conflicting requests 2024-04-06T02:39:21.714 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.714 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 14: conflicting requests 2024-04-06T02:39:21.714 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.715 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 15: conflicting requests 2024-04-06T02:39:21.715 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.715 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 16: conflicting requests 2024-04-06T02:39:21.715 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.715 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 17: conflicting requests 2024-04-06T02:39:21.715 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.715 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 18: conflicting requests 2024-04-06T02:39:21.715 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.715 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 19: conflicting requests 2024-04-06T02:39:21.715 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.715 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 20: conflicting requests 2024-04-06T02:39:21.716 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.716 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 21: conflicting requests 2024-04-06T02:39:21.716 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.716 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 22: conflicting requests 2024-04-06T02:39:21.716 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.716 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 23: conflicting requests 2024-04-06T02:39:21.716 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.716 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 24: conflicting requests 2024-04-06T02:39:21.716 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.716 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 25: conflicting requests 2024-04-06T02:39:21.716 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.716 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 26: conflicting requests 2024-04-06T02:39:21.716 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.717 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 27: conflicting requests 2024-04-06T02:39:21.717 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.717 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 28: conflicting requests 2024-04-06T02:39:21.717 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.717 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 29: conflicting requests 2024-04-06T02:39:21.717 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.717 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 30: conflicting requests 2024-04-06T02:39:21.717 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.717 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 31: conflicting requests 2024-04-06T02:39:21.717 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.717 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 32: conflicting requests 2024-04-06T02:39:21.718 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.718 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 33: conflicting requests 2024-04-06T02:39:21.718 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.718 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 34: conflicting requests 2024-04-06T02:39:21.718 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.718 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 35: conflicting requests 2024-04-06T02:39:21.718 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.718 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 36: conflicting requests 2024-04-06T02:39:21.718 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.718 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 37: conflicting requests 2024-04-06T02:39:21.718 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.718 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.718 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.719 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:39:21.719 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.719 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.719 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.719 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.719 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 39: conflicting requests 2024-04-06T02:39:21.719 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.719 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.719 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.719 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.719 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.719 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.719 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:39:21.720 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.720 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.720 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.720 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.720 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:39:21.720 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.720 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.720 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.720 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.720 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:39:21.720 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.720 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.720 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.720 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.720 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:39:21.721 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.721 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.721 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.721 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.721 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:39:21.721 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.721 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.721 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.721 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.721 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:39:21.721 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.721 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.721 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.722 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:39:21.787 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-04-06T02:39:21.789 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:39:21.789 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-04-06T02:39:21.789 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:39:21.790 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-04-06T02:39:21.790 INFO:teuthology.orchestra.run.smithi184.stdout: ceph x86_64 2:18.2.1-0.el9 ceph 6.4 k 2024-04-06T02:39:21.790 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-04-06T02:39:21.790 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mds x86_64 2:18.2.1-0.el9 ceph 2.1 M 2024-04-06T02:39:21.790 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr x86_64 2:18.2.1-0.el9 ceph 1.4 M 2024-04-06T02:39:21.790 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-modules-core noarch 2:18.2.1-0.el9 ceph-noarch 242 k 2024-04-06T02:39:21.790 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mon x86_64 2:18.2.1-0.el9 ceph 4.4 M 2024-04-06T02:39:21.790 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-osd x86_64 2:18.2.1-0.el9 ceph 18 M 2024-04-06T02:39:21.790 INFO:teuthology.orchestra.run.smithi184.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-06T02:39:21.790 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-06T02:39:21.790 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-06T02:39:21.790 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-06T02:39:21.791 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-06T02:39:21.791 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-06T02:39:21.791 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-06T02:39:21.791 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-06T02:39:21.791 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-06T02:39:21.791 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-06T02:39:21.791 INFO:teuthology.orchestra.run.smithi184.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-06T02:39:21.791 INFO:teuthology.orchestra.run.smithi184.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-06T02:39:21.791 INFO:teuthology.orchestra.run.smithi184.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-06T02:39:21.791 INFO:teuthology.orchestra.run.smithi184.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-06T02:39:21.791 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-06T02:39:21.791 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-06T02:39:21.791 INFO:teuthology.orchestra.run.smithi184.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-06T02:39:21.791 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-06T02:39:21.792 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-06T02:39:21.792 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-06T02:39:21.792 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-06T02:39:21.792 INFO:teuthology.orchestra.run.smithi184.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-06T02:39:21.792 INFO:teuthology.orchestra.run.smithi184.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-06T02:39:21.792 INFO:teuthology.orchestra.run.smithi184.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-06T02:39:21.792 INFO:teuthology.orchestra.run.smithi184.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-06T02:39:21.792 INFO:teuthology.orchestra.run.smithi184.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-06T02:39:21.792 INFO:teuthology.orchestra.run.smithi184.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-06T02:39:21.792 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:21.792 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-04-06T02:39:21.792 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:39:21.792 INFO:teuthology.orchestra.run.smithi184.stdout:Install 33 Packages 2024-04-06T02:39:21.793 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:21.793 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 30 M 2024-04-06T02:39:21.793 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 110 M 2024-04-06T02:39:21.793 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-04-06T02:39:21.862 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:55 ago on Sat 06 Apr 2024 02:38:26 AM UTC. 2024-04-06T02:39:22.148 INFO:teuthology.orchestra.run.smithi105.stderr:Modular dependency problems: 2024-04-06T02:39:22.148 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-06T02:39:22.148 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 1: conflicting requests 2024-04-06T02:39:22.149 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.149 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 2: conflicting requests 2024-04-06T02:39:22.149 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.149 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 3: conflicting requests 2024-04-06T02:39:22.149 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.149 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 4: conflicting requests 2024-04-06T02:39:22.149 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.149 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 5: conflicting requests 2024-04-06T02:39:22.149 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.149 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 6: conflicting requests 2024-04-06T02:39:22.149 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.149 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 7: conflicting requests 2024-04-06T02:39:22.149 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.150 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 8: conflicting requests 2024-04-06T02:39:22.150 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.150 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 9: conflicting requests 2024-04-06T02:39:22.150 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.150 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 10: conflicting requests 2024-04-06T02:39:22.150 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.150 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 11: conflicting requests 2024-04-06T02:39:22.150 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.150 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 12: conflicting requests 2024-04-06T02:39:22.150 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.150 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 13: conflicting requests 2024-04-06T02:39:22.150 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.150 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 14: conflicting requests 2024-04-06T02:39:22.151 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.151 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 15: conflicting requests 2024-04-06T02:39:22.151 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.151 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 16: conflicting requests 2024-04-06T02:39:22.151 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.151 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 17: conflicting requests 2024-04-06T02:39:22.151 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.151 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 18: conflicting requests 2024-04-06T02:39:22.151 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.151 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 19: conflicting requests 2024-04-06T02:39:22.151 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.151 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 20: conflicting requests 2024-04-06T02:39:22.151 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.151 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 21: conflicting requests 2024-04-06T02:39:22.151 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.152 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 22: conflicting requests 2024-04-06T02:39:22.152 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.152 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 23: conflicting requests 2024-04-06T02:39:22.152 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.152 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 24: conflicting requests 2024-04-06T02:39:22.152 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.152 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 25: conflicting requests 2024-04-06T02:39:22.152 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.152 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 26: conflicting requests 2024-04-06T02:39:22.152 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.152 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 27: conflicting requests 2024-04-06T02:39:22.152 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.152 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 28: conflicting requests 2024-04-06T02:39:22.152 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.153 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 29: conflicting requests 2024-04-06T02:39:22.153 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.153 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 30: conflicting requests 2024-04-06T02:39:22.153 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.153 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 31: conflicting requests 2024-04-06T02:39:22.153 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.153 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 32: conflicting requests 2024-04-06T02:39:22.153 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.153 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 33: conflicting requests 2024-04-06T02:39:22.153 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.153 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 34: conflicting requests 2024-04-06T02:39:22.153 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.153 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 35: conflicting requests 2024-04-06T02:39:22.154 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.154 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 36: conflicting requests 2024-04-06T02:39:22.154 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.154 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 37: conflicting requests 2024-04-06T02:39:22.154 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.154 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.154 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.154 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:39:22.154 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.154 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.154 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.154 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.154 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 39: conflicting requests 2024-04-06T02:39:22.154 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.154 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.155 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.155 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.155 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.155 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.155 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:39:22.155 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.155 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.155 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.155 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.155 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:39:22.155 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.155 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.155 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.156 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.156 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:39:22.156 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.156 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.156 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.156 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.156 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:39:22.156 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.156 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.156 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.156 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.156 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:39:22.156 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.156 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.156 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.157 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.157 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:39:22.157 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.157 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.157 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.157 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:39:22.214 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-06T02:39:22.215 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:39:22.216 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2024-04-06T02:39:22.216 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:39:22.216 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-06T02:39:22.216 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm noarch 2:18.2.1-0.el9 ceph-noarch 221 k 2024-04-06T02:39:22.216 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:22.216 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-06T02:39:22.216 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:39:22.216 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-04-06T02:39:22.216 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:22.216 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 221 k 2024-04-06T02:39:22.216 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 213 k 2024-04-06T02:39:22.216 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-06T02:39:22.328 INFO:teuthology.orchestra.run.smithi105.stdout:cephadm-18.2.1-0.el9.noarch.rpm 1.9 MB/s | 221 kB 00:00 2024-04-06T02:39:22.329 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:39:22.329 INFO:teuthology.orchestra.run.smithi105.stdout:Total 1.9 MB/s | 221 kB 00:00 2024-04-06T02:39:22.329 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-06T02:39:22.334 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-06T02:39:22.334 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-06T02:39:22.336 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-06T02:39:22.337 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-06T02:39:22.413 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-06T02:39:22.609 INFO:teuthology.orchestra.run.smithi184.stdout:(1/33): ceph-18.2.1-0.el9.x86_64.rpm 59 kB/s | 6.4 kB 00:00 2024-04-06T02:39:22.651 INFO:teuthology.orchestra.run.smithi184.stdout:(2/33): ceph-mgr-18.2.1-0.el9.x86_64.rpm 9.7 MB/s | 1.4 MB 00:00 2024-04-06T02:39:22.693 INFO:teuthology.orchestra.run.smithi184.stdout:(3/33): ceph-mds-18.2.1-0.el9.x86_64.rpm 11 MB/s | 2.1 MB 00:00 2024-04-06T02:39:22.718 INFO:teuthology.orchestra.run.smithi184.stdout:(4/33): ceph-mgr-modules-core-18.2.1-0.el9.noar 9.5 MB/s | 242 kB 00:00 2024-04-06T02:39:22.794 INFO:teuthology.orchestra.run.smithi184.stdout:(5/33): ceph-mon-18.2.1-0.el9.x86_64.rpm 24 MB/s | 4.4 MB 00:00 2024-04-06T02:39:22.801 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-06T02:39:22.888 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-06T02:39:23.011 INFO:teuthology.orchestra.run.smithi184.stdout:(6/33): ceph-osd-18.2.1-0.el9.x86_64.rpm 49 MB/s | 18 MB 00:00 2024-04-06T02:39:23.255 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-06T02:39:23.547 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-06T02:39:23.547 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:23.547 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-06T02:39:23.548 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm-2:18.2.1-0.el9.noarch 2024-04-06T02:39:23.548 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:23.548 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-06T02:39:23.744 INFO:teuthology.orchestra.run.smithi184.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 145 kB/s | 106 kB 00:00 2024-04-06T02:39:23.764 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-immutable-object-cache 2024-04-06T02:39:23.819 INFO:teuthology.orchestra.run.smithi184.stdout:(8/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 230 kB/s | 253 kB 00:01 2024-04-06T02:39:24.363 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:58 ago on Sat 06 Apr 2024 02:38:26 AM UTC. 2024-04-06T02:39:24.544 INFO:teuthology.orchestra.run.smithi184.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 169 kB/s | 135 kB 00:00 2024-04-06T02:39:24.578 INFO:teuthology.orchestra.run.smithi184.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 165 kB/s | 125 kB 00:00 2024-04-06T02:39:24.648 INFO:teuthology.orchestra.run.smithi105.stderr:Modular dependency problems: 2024-04-06T02:39:24.648 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-06T02:39:24.649 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 1: conflicting requests 2024-04-06T02:39:24.649 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.649 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 2: conflicting requests 2024-04-06T02:39:24.649 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.649 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 3: conflicting requests 2024-04-06T02:39:24.649 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.649 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 4: conflicting requests 2024-04-06T02:39:24.649 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.649 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 5: conflicting requests 2024-04-06T02:39:24.649 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.649 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 6: conflicting requests 2024-04-06T02:39:24.649 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.649 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 7: conflicting requests 2024-04-06T02:39:24.650 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.650 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 8: conflicting requests 2024-04-06T02:39:24.650 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.650 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 9: conflicting requests 2024-04-06T02:39:24.650 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.650 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 10: conflicting requests 2024-04-06T02:39:24.650 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.650 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 11: conflicting requests 2024-04-06T02:39:24.650 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.650 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 12: conflicting requests 2024-04-06T02:39:24.650 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.650 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 13: conflicting requests 2024-04-06T02:39:24.650 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.650 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 14: conflicting requests 2024-04-06T02:39:24.651 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.651 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 15: conflicting requests 2024-04-06T02:39:24.651 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.651 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 16: conflicting requests 2024-04-06T02:39:24.651 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.651 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 17: conflicting requests 2024-04-06T02:39:24.651 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.651 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 18: conflicting requests 2024-04-06T02:39:24.651 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.651 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 19: conflicting requests 2024-04-06T02:39:24.651 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.651 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 20: conflicting requests 2024-04-06T02:39:24.651 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.652 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 21: conflicting requests 2024-04-06T02:39:24.652 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.652 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 22: conflicting requests 2024-04-06T02:39:24.652 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.652 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 23: conflicting requests 2024-04-06T02:39:24.652 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.652 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 24: conflicting requests 2024-04-06T02:39:24.652 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.652 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 25: conflicting requests 2024-04-06T02:39:24.652 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.652 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 26: conflicting requests 2024-04-06T02:39:24.652 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.652 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 27: conflicting requests 2024-04-06T02:39:24.652 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.652 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 28: conflicting requests 2024-04-06T02:39:24.653 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.653 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 29: conflicting requests 2024-04-06T02:39:24.653 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.653 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 30: conflicting requests 2024-04-06T02:39:24.653 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.653 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 31: conflicting requests 2024-04-06T02:39:24.653 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.653 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 32: conflicting requests 2024-04-06T02:39:24.653 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.653 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 33: conflicting requests 2024-04-06T02:39:24.653 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.653 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 34: conflicting requests 2024-04-06T02:39:24.653 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.653 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 35: conflicting requests 2024-04-06T02:39:24.654 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.654 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 36: conflicting requests 2024-04-06T02:39:24.654 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.654 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 37: conflicting requests 2024-04-06T02:39:24.654 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.654 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.654 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.654 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:39:24.654 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.654 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.654 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.654 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.654 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 39: conflicting requests 2024-04-06T02:39:24.655 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.655 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.655 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.655 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.655 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.655 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.655 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:39:24.655 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.655 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.655 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.655 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.655 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:39:24.655 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.655 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.656 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.656 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.656 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:39:24.656 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.656 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.656 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.656 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.656 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:39:24.656 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.656 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.656 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.656 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.656 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:39:24.656 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.657 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.657 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.657 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.657 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:39:24.657 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.657 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.657 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.657 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:39:24.716 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-06T02:39:24.717 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:39:24.717 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2024-04-06T02:39:24.717 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:39:24.717 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-06T02:39:24.717 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-0.el9 ceph 142 k 2024-04-06T02:39:24.717 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:24.717 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-06T02:39:24.717 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:39:24.717 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-04-06T02:39:24.717 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:24.717 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 142 k 2024-04-06T02:39:24.718 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 395 k 2024-04-06T02:39:24.718 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-06T02:39:24.762 INFO:teuthology.orchestra.run.smithi184.stdout:(11/33): python3-mako-1.1.4-6.el9.noarch.rpm 941 kB/s | 172 kB 00:00 2024-04-06T02:39:24.815 INFO:teuthology.orchestra.run.smithi105.stdout:ceph-immutable-object-cache-18.2.1-0.el9.x86_64 1.4 MB/s | 142 kB 00:00 2024-04-06T02:39:24.815 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:39:24.815 INFO:teuthology.orchestra.run.smithi105.stdout:Total 1.4 MB/s | 142 kB 00:00 2024-04-06T02:39:24.816 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-06T02:39:24.823 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-06T02:39:24.823 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-06T02:39:24.845 INFO:teuthology.orchestra.run.smithi184.stdout:(12/33): python3-markupsafe-1.1.1-12.el9.x86_64 416 kB/s | 35 kB 00:00 2024-04-06T02:39:24.876 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-06T02:39:24.876 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-06T02:39:24.921 INFO:teuthology.orchestra.run.smithi184.stdout:(13/33): python3-pytz-2021.1-5.el9.noarch.rpm 678 kB/s | 51 kB 00:00 2024-04-06T02:39:25.004 INFO:teuthology.orchestra.run.smithi184.stdout:(14/33): python3-toml-0.10.2-6.el9.noarch.rpm 506 kB/s | 42 kB 00:00 2024-04-06T02:39:25.020 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-06T02:39:25.082 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:39:25.107 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:39:25.108 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-06T02:39:25.108 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-06T02:39:25.108 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:25.246 INFO:teuthology.orchestra.run.smithi184.stdout:(15/33): python3-cryptography-36.0.1-4.el9.x86_ 512 kB/s | 1.2 MB 00:02 2024-04-06T02:39:25.405 INFO:teuthology.orchestra.run.smithi184.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 108 kB/s | 43 kB 00:00 2024-04-06T02:39:25.772 INFO:teuthology.orchestra.run.smithi184.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 978 kB/s | 358 kB 00:00 2024-04-06T02:39:25.797 INFO:teuthology.orchestra.run.smithi184.stdout:(18/33): python3-urllib3-1.26.5-5.el9.noarch.rp 171 kB/s | 215 kB 00:01 2024-04-06T02:39:25.822 INFO:teuthology.orchestra.run.smithi184.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 298 kB/s | 172 kB 00:00 2024-04-06T02:39:25.873 INFO:teuthology.orchestra.run.smithi184.stdout:(20/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 106 kB/s | 11 kB 00:00 2024-04-06T02:39:25.931 INFO:teuthology.orchestra.run.smithi184.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 213 kB/s | 23 kB 00:00 2024-04-06T02:39:25.998 INFO:teuthology.orchestra.run.smithi184.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 196 kB/s | 19 kB 00:00 2024-04-06T02:39:26.040 INFO:teuthology.orchestra.run.smithi184.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2024-04-06T02:39:26.099 INFO:teuthology.orchestra.run.smithi184.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 461 kB/s | 46 kB 00:00 2024-04-06T02:39:26.149 INFO:teuthology.orchestra.run.smithi184.stdout:(25/33): python3-jaraco-classes-3.2.1-5.el9.noa 50 kB/s | 18 kB 00:00 2024-04-06T02:39:26.174 INFO:teuthology.orchestra.run.smithi184.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 589 kB/s | 79 kB 00:00 2024-04-06T02:39:26.177 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:39:26.177 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:26.177 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-06T02:39:26.177 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 2024-04-06T02:39:26.177 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:26.177 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-06T02:39:26.233 INFO:teuthology.orchestra.run.smithi184.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-04-06T02:39:26.266 INFO:teuthology.orchestra.run.smithi184.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 140 kB/s | 16 kB 00:00 2024-04-06T02:39:26.300 INFO:teuthology.orchestra.run.smithi184.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 716 kB/s | 90 kB 00:00 2024-04-06T02:39:26.334 INFO:teuthology.orchestra.run.smithi184.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 355 kB/s | 36 kB 00:00 2024-04-06T02:39:26.434 INFO:teuthology.orchestra.run.smithi184.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2024-04-06T02:39:26.458 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr 2024-04-06T02:39:26.559 INFO:teuthology.orchestra.run.smithi184.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.6 MB/s | 427 kB 00:00 2024-04-06T02:39:26.584 INFO:teuthology.orchestra.run.smithi184.stdout:(33/33): python3-webob-1.8.7-6.el9.noarch.rpm 723 kB/s | 230 kB 00:00 2024-04-06T02:39:26.590 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:39:26.590 INFO:teuthology.orchestra.run.smithi184.stdout:Total 6.2 MB/s | 30 MB 00:04 2024-04-06T02:39:26.709 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-04-06T02:39:26.747 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-04-06T02:39:26.747 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-04-06T02:39:27.055 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:01 ago on Sat 06 Apr 2024 02:38:26 AM UTC. 2024-04-06T02:39:27.110 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-04-06T02:39:27.111 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-04-06T02:39:27.340 INFO:teuthology.orchestra.run.smithi105.stderr:Modular dependency problems: 2024-04-06T02:39:27.341 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-06T02:39:27.341 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 1: conflicting requests 2024-04-06T02:39:27.341 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.341 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 2: conflicting requests 2024-04-06T02:39:27.341 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.341 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 3: conflicting requests 2024-04-06T02:39:27.341 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.341 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 4: conflicting requests 2024-04-06T02:39:27.341 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.341 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 5: conflicting requests 2024-04-06T02:39:27.341 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.341 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 6: conflicting requests 2024-04-06T02:39:27.341 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.342 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 7: conflicting requests 2024-04-06T02:39:27.342 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.342 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 8: conflicting requests 2024-04-06T02:39:27.342 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.342 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 9: conflicting requests 2024-04-06T02:39:27.342 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.342 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 10: conflicting requests 2024-04-06T02:39:27.342 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.342 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 11: conflicting requests 2024-04-06T02:39:27.342 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.342 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 12: conflicting requests 2024-04-06T02:39:27.342 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.342 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 13: conflicting requests 2024-04-06T02:39:27.343 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.343 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 14: conflicting requests 2024-04-06T02:39:27.343 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.343 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 15: conflicting requests 2024-04-06T02:39:27.343 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.343 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 16: conflicting requests 2024-04-06T02:39:27.343 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.343 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 17: conflicting requests 2024-04-06T02:39:27.343 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.343 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 18: conflicting requests 2024-04-06T02:39:27.343 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.343 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 19: conflicting requests 2024-04-06T02:39:27.343 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.344 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 20: conflicting requests 2024-04-06T02:39:27.344 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.344 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 21: conflicting requests 2024-04-06T02:39:27.344 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.344 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 22: conflicting requests 2024-04-06T02:39:27.344 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.344 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 23: conflicting requests 2024-04-06T02:39:27.344 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.344 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 24: conflicting requests 2024-04-06T02:39:27.344 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.344 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 25: conflicting requests 2024-04-06T02:39:27.344 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.344 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 26: conflicting requests 2024-04-06T02:39:27.344 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.345 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 27: conflicting requests 2024-04-06T02:39:27.345 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.345 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 28: conflicting requests 2024-04-06T02:39:27.345 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.345 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 29: conflicting requests 2024-04-06T02:39:27.345 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.345 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 30: conflicting requests 2024-04-06T02:39:27.345 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.345 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 31: conflicting requests 2024-04-06T02:39:27.345 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.345 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 32: conflicting requests 2024-04-06T02:39:27.345 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.345 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 33: conflicting requests 2024-04-06T02:39:27.346 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.346 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 34: conflicting requests 2024-04-06T02:39:27.346 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.346 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 35: conflicting requests 2024-04-06T02:39:27.346 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.346 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 36: conflicting requests 2024-04-06T02:39:27.346 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.346 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 37: conflicting requests 2024-04-06T02:39:27.346 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.346 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.346 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.346 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:39:27.346 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.347 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.347 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.347 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.347 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 39: conflicting requests 2024-04-06T02:39:27.347 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.347 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.347 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.347 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.347 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.347 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.347 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:39:27.347 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.347 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.348 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.348 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.348 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:39:27.348 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.348 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.348 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.348 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.348 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:39:27.348 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.348 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.348 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.348 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.348 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:39:27.349 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.349 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.349 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.349 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.349 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:39:27.349 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.349 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.349 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.349 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.349 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:39:27.349 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.349 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.349 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.350 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:39:27.350 INFO:teuthology.orchestra.run.smithi105.stdout:Package ceph-mgr-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-06T02:39:27.408 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-06T02:39:27.409 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-04-06T02:39:27.409 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-06T02:39:27.486 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-dashboard 2024-04-06T02:39:28.063 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:02 ago on Sat 06 Apr 2024 02:38:26 AM UTC. 2024-04-06T02:39:28.348 INFO:teuthology.orchestra.run.smithi105.stderr:Modular dependency problems: 2024-04-06T02:39:28.348 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-06T02:39:28.348 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 1: conflicting requests 2024-04-06T02:39:28.348 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.349 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 2: conflicting requests 2024-04-06T02:39:28.349 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.349 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 3: conflicting requests 2024-04-06T02:39:28.349 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.349 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 4: conflicting requests 2024-04-06T02:39:28.349 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.349 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 5: conflicting requests 2024-04-06T02:39:28.349 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.349 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 6: conflicting requests 2024-04-06T02:39:28.349 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.349 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 7: conflicting requests 2024-04-06T02:39:28.349 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.349 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 8: conflicting requests 2024-04-06T02:39:28.349 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.350 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 9: conflicting requests 2024-04-06T02:39:28.350 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.350 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 10: conflicting requests 2024-04-06T02:39:28.350 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.350 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 11: conflicting requests 2024-04-06T02:39:28.350 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.350 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 12: conflicting requests 2024-04-06T02:39:28.350 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.350 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 13: conflicting requests 2024-04-06T02:39:28.350 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.350 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 14: conflicting requests 2024-04-06T02:39:28.350 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.351 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 15: conflicting requests 2024-04-06T02:39:28.351 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.351 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 16: conflicting requests 2024-04-06T02:39:28.351 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.351 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 17: conflicting requests 2024-04-06T02:39:28.351 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.355 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 18: conflicting requests 2024-04-06T02:39:28.355 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.355 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 19: conflicting requests 2024-04-06T02:39:28.355 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.356 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 20: conflicting requests 2024-04-06T02:39:28.356 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.356 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 21: conflicting requests 2024-04-06T02:39:28.356 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.356 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 22: conflicting requests 2024-04-06T02:39:28.357 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.357 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 23: conflicting requests 2024-04-06T02:39:28.357 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.357 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 24: conflicting requests 2024-04-06T02:39:28.357 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.357 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 25: conflicting requests 2024-04-06T02:39:28.358 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.358 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 26: conflicting requests 2024-04-06T02:39:28.358 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.358 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 27: conflicting requests 2024-04-06T02:39:28.358 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.358 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 28: conflicting requests 2024-04-06T02:39:28.358 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.358 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 29: conflicting requests 2024-04-06T02:39:28.358 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.358 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 30: conflicting requests 2024-04-06T02:39:28.358 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.359 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 31: conflicting requests 2024-04-06T02:39:28.359 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.359 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 32: conflicting requests 2024-04-06T02:39:28.359 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.359 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 33: conflicting requests 2024-04-06T02:39:28.359 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.359 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 34: conflicting requests 2024-04-06T02:39:28.359 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.359 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 35: conflicting requests 2024-04-06T02:39:28.359 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.359 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 36: conflicting requests 2024-04-06T02:39:28.359 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.359 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 37: conflicting requests 2024-04-06T02:39:28.360 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.360 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.360 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.360 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:39:28.360 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.360 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.360 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.360 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.360 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 39: conflicting requests 2024-04-06T02:39:28.360 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.360 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.360 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.360 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.360 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.361 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.361 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:39:28.361 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.361 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.361 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.361 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.361 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:39:28.361 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.361 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.361 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.361 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.361 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:39:28.362 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.362 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.362 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.362 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.362 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:39:28.362 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.362 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.362 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.362 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.362 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:39:28.362 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.362 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.362 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.362 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.363 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:39:28.363 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.363 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.363 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.363 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:39:28.363 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-04-06T02:39:28.416 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-06T02:39:28.418 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:39:28.418 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-04-06T02:39:28.418 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:39:28.419 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-06T02:39:28.419 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard noarch 2:18.2.1-0.el9 ceph-noarch 1.8 M 2024-04-06T02:39:28.419 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-04-06T02:39:28.419 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards noarch 2:18.2.1-0.el9 ceph-noarch 23 k 2024-04-06T02:39:28.419 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts noarch 2:18.2.1-0.el9 ceph-noarch 15 k 2024-04-06T02:39:28.419 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-04-06T02:39:28.420 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-06T02:39:28.420 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-06T02:39:28.420 INFO:teuthology.orchestra.run.smithi105.stdout:Installing weak dependencies: 2024-04-06T02:39:28.420 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-04-06T02:39:28.420 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:28.420 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-06T02:39:28.421 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:39:28.421 INFO:teuthology.orchestra.run.smithi105.stdout:Install 7 Packages 2024-04-06T02:39:28.421 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:28.421 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 2.1 M 2024-04-06T02:39:28.421 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 8.5 M 2024-04-06T02:39:28.421 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-06T02:39:28.428 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-06T02:39:28.461 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-06T02:39:28.493 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-06T02:39:28.587 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-06T02:39:28.664 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-06T02:39:28.705 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-06T02:39:28.748 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-06T02:39:28.778 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-04-06T02:39:28.793 INFO:teuthology.orchestra.run.smithi105.stdout:(1/7): ceph-grafana-dashboards-18.2.1-0.el9.noa 166 kB/s | 23 kB 00:00 2024-04-06T02:39:28.812 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-04-06T02:39:28.827 INFO:teuthology.orchestra.run.smithi105.stdout:(2/7): ceph-prometheus-alerts-18.2.1-0.el9.noar 85 kB/s | 15 kB 00:00 2024-04-06T02:39:28.846 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-04-06T02:39:28.869 INFO:teuthology.orchestra.run.smithi105.stdout:(3/7): ceph-mgr-dashboard-18.2.1-0.el9.noarch.r 8.2 MB/s | 1.8 MB 00:00 2024-04-06T02:39:28.886 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-04-06T02:39:28.921 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-04-06T02:39:28.955 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-04-06T02:39:29.001 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-04-06T02:39:29.036 INFO:teuthology.orchestra.run.smithi105.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 37 kB/s | 9.0 kB 00:00 2024-04-06T02:39:29.069 INFO:teuthology.orchestra.run.smithi105.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 169 kB/s | 41 kB 00:00 2024-04-06T02:39:29.075 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-04-06T02:39:29.094 INFO:teuthology.orchestra.run.smithi105.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 136 kB/s | 31 kB 00:00 2024-04-06T02:39:29.161 INFO:teuthology.orchestra.run.smithi105.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-04-06T02:39:29.165 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:39:29.165 INFO:teuthology.orchestra.run.smithi105.stdout:Total 2.8 MB/s | 2.1 MB 00:00 2024-04-06T02:39:29.165 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-06T02:39:29.173 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-06T02:39:29.173 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-06T02:39:29.222 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-04-06T02:39:29.240 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-06T02:39:29.240 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-06T02:39:29.302 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-04-06T02:39:29.355 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-04-06T02:39:29.393 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-04-06T02:39:29.442 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-04-06T02:39:29.499 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-04-06T02:39:29.550 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-06T02:39:29.621 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-04-06T02:39:29.647 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-04-06T02:39:29.693 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-04-06T02:39:29.708 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-04-06T02:39:29.725 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-04-06T02:39:29.745 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 5/7 2024-04-06T02:39:29.756 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-04-06T02:39:29.826 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-04-06T02:39:29.915 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-04-06T02:39:29.975 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 6/7 2024-04-06T02:39:29.977 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-04-06T02:39:30.007 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-04-06T02:39:30.097 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-04-06T02:39:30.249 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 28/33 2024-04-06T02:39:30.282 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-04-06T02:39:30.307 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-04-06T02:39:30.307 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-06T02:39:30.307 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-06T02:39:30.307 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-06T02:39:30.307 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-06T02:39:30.307 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:30.522 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-04-06T02:39:30.522 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 1/7 2024-04-06T02:39:30.522 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2/7 2024-04-06T02:39:30.522 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 3/7 2024-04-06T02:39:30.522 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-04-06T02:39:30.522 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-04-06T02:39:30.522 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-04-06T02:39:30.831 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-04-06T02:39:30.832 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:30.832 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-06T02:39:30.832 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 2024-04-06T02:39:30.832 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2024-04-06T02:39:30.832 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 2024-04-06T02:39:30.832 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-04-06T02:39:30.832 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-04-06T02:39:30.832 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-06T02:39:30.832 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-06T02:39:30.833 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:30.833 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-06T02:39:31.074 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-06T02:39:31.663 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:05 ago on Sat 06 Apr 2024 02:38:26 AM UTC. 2024-04-06T02:39:31.833 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-04-06T02:39:31.861 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-04-06T02:39:31.861 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-06T02:39:31.862 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-06T02:39:31.862 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-06T02:39:31.862 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-06T02:39:31.862 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:31.948 INFO:teuthology.orchestra.run.smithi105.stderr:Modular dependency problems: 2024-04-06T02:39:31.948 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-06T02:39:31.948 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 1: conflicting requests 2024-04-06T02:39:31.948 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.948 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 2: conflicting requests 2024-04-06T02:39:31.948 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.948 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 3: conflicting requests 2024-04-06T02:39:31.949 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.949 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 4: conflicting requests 2024-04-06T02:39:31.949 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.949 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 5: conflicting requests 2024-04-06T02:39:31.949 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.949 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 6: conflicting requests 2024-04-06T02:39:31.949 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.949 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 7: conflicting requests 2024-04-06T02:39:31.949 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.949 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 8: conflicting requests 2024-04-06T02:39:31.949 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.949 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 9: conflicting requests 2024-04-06T02:39:31.949 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.949 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 10: conflicting requests 2024-04-06T02:39:31.949 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.950 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 11: conflicting requests 2024-04-06T02:39:31.950 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.950 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 12: conflicting requests 2024-04-06T02:39:31.950 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.950 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 13: conflicting requests 2024-04-06T02:39:31.950 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.950 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 14: conflicting requests 2024-04-06T02:39:31.950 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.950 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 15: conflicting requests 2024-04-06T02:39:31.950 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.950 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 16: conflicting requests 2024-04-06T02:39:31.950 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.950 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 17: conflicting requests 2024-04-06T02:39:31.950 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.951 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 18: conflicting requests 2024-04-06T02:39:31.951 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.951 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 19: conflicting requests 2024-04-06T02:39:31.951 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.951 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 20: conflicting requests 2024-04-06T02:39:31.951 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.951 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 21: conflicting requests 2024-04-06T02:39:31.951 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.951 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 22: conflicting requests 2024-04-06T02:39:31.951 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.951 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 23: conflicting requests 2024-04-06T02:39:31.951 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.951 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 24: conflicting requests 2024-04-06T02:39:31.951 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.952 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 25: conflicting requests 2024-04-06T02:39:31.952 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.952 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 26: conflicting requests 2024-04-06T02:39:31.952 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.952 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 27: conflicting requests 2024-04-06T02:39:31.952 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.952 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 28: conflicting requests 2024-04-06T02:39:31.952 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.952 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 29: conflicting requests 2024-04-06T02:39:31.952 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.952 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 30: conflicting requests 2024-04-06T02:39:31.952 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.952 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 31: conflicting requests 2024-04-06T02:39:31.952 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.952 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 32: conflicting requests 2024-04-06T02:39:31.952 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.953 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 33: conflicting requests 2024-04-06T02:39:31.953 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.953 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 34: conflicting requests 2024-04-06T02:39:31.953 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.953 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 35: conflicting requests 2024-04-06T02:39:31.953 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.953 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 36: conflicting requests 2024-04-06T02:39:31.953 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.953 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 37: conflicting requests 2024-04-06T02:39:31.953 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.953 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.953 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.953 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:39:31.953 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.953 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.954 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.954 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.954 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 39: conflicting requests 2024-04-06T02:39:31.954 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.954 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.954 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.954 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.954 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.954 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.954 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:39:31.954 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.954 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.954 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.955 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.955 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:39:31.955 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.955 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.955 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.955 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.955 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:39:31.955 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.955 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.955 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.955 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.955 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:39:31.955 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.955 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.955 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.955 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.956 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:39:31.956 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.956 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.956 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.956 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.956 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:39:31.956 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.956 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.956 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:39:31.956 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:39:32.016 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-06T02:39:32.017 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:39:32.017 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-04-06T02:39:32.017 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:39:32.017 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-06T02:39:32.018 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-0.el9 ceph-noarch 7.4 M 2024-04-06T02:39:32.018 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-04-06T02:39:32.018 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-06T02:39:32.018 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-06T02:39:32.018 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-06T02:39:32.018 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-06T02:39:32.018 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-06T02:39:32.018 INFO:teuthology.orchestra.run.smithi105.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-06T02:39:32.018 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-06T02:39:32.018 INFO:teuthology.orchestra.run.smithi105.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-06T02:39:32.018 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-06T02:39:32.019 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-06T02:39:32.019 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-06T02:39:32.019 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:32.019 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-06T02:39:32.019 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:39:32.019 INFO:teuthology.orchestra.run.smithi105.stdout:Install 12 Packages 2024-04-06T02:39:32.019 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:32.019 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 38 M 2024-04-06T02:39:32.020 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 209 M 2024-04-06T02:39:32.020 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-06T02:39:32.326 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-04-06T02:39:32.351 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-04-06T02:39:32.351 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-06T02:39:32.351 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-06T02:39:32.351 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-06T02:39:32.351 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-06T02:39:32.351 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:32.568 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-04-06T02:39:32.592 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-04-06T02:39:32.592 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-06T02:39:32.592 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-06T02:39:32.592 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-06T02:39:32.592 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-06T02:39:32.592 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:32.651 INFO:teuthology.orchestra.run.smithi105.stdout:(1/12): ceph-mgr-diskprediction-local-18.2.1-0. 22 MB/s | 7.4 MB 00:00 2024-04-06T02:39:32.661 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-2:18.2.1-0.el9.x86_64 33/33 2024-04-06T02:39:32.835 INFO:teuthology.orchestra.run.smithi105.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 240 kB/s | 33 kB 00:00 2024-04-06T02:39:33.185 INFO:teuthology.orchestra.run.smithi105.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.5 MB/s | 3.0 MB 00:00 2024-04-06T02:39:33.252 INFO:teuthology.orchestra.run.smithi105.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 273 kB/s | 18 kB 00:00 2024-04-06T02:39:33.310 INFO:teuthology.orchestra.run.smithi105.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 605 kB/s | 35 kB 00:00 2024-04-06T02:39:33.335 INFO:teuthology.orchestra.run.smithi105.stdout:(6/12): libquadmath-11.4.1-3.el9.x86_64.rpm 192 kB/s | 195 kB 00:01 2024-04-06T02:39:33.628 INFO:teuthology.orchestra.run.smithi105.stdout:(7/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2024-04-06T02:39:33.653 INFO:teuthology.orchestra.run.smithi105.stdout:(8/12): python3-devel-3.9.17-2.el9.x86_64.rpm 774 kB/s | 245 kB 00:00 2024-04-06T02:39:33.787 INFO:teuthology.orchestra.run.smithi105.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.8 MB/s | 250 kB 00:00 2024-04-06T02:39:33.904 INFO:teuthology.orchestra.run.smithi105.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 5.1 MB 00:00 2024-04-06T02:39:34.112 INFO:teuthology.orchestra.run.smithi105.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 448 kB/s | 803 kB 00:01 2024-04-06T02:39:34.509 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-2:18.2.1-0.el9.x86_64 33/33 2024-04-06T02:39:34.509 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-2:18.2.1-0.el9.x86_64 1/33 2024-04-06T02:39:34.510 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mds-2:18.2.1-0.el9.x86_64 2/33 2024-04-06T02:39:34.510 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-2:18.2.1-0.el9.x86_64 3/33 2024-04-06T02:39:34.510 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mon-2:18.2.1-0.el9.x86_64 4/33 2024-04-06T02:39:34.510 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-osd-2:18.2.1-0.el9.x86_64 5/33 2024-04-06T02:39:34.510 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 6/33 2024-04-06T02:39:34.510 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-04-06T02:39:34.510 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-04-06T02:39:34.510 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-04-06T02:39:34.510 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-04-06T02:39:34.510 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-04-06T02:39:34.510 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-04-06T02:39:34.511 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-04-06T02:39:34.511 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-04-06T02:39:34.511 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-04-06T02:39:34.511 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-06T02:39:34.511 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-06T02:39:34.511 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-06T02:39:34.511 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-06T02:39:34.511 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-06T02:39:34.511 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-06T02:39:34.511 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-06T02:39:34.511 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-06T02:39:34.511 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-06T02:39:34.512 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-06T02:39:34.512 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-06T02:39:34.513 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-06T02:39:34.513 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-06T02:39:34.513 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-06T02:39:34.513 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-06T02:39:34.513 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-06T02:39:34.513 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-06T02:39:35.584 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-06T02:39:35.584 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:35.585 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-04-06T02:39:35.585 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-2:18.2.1-0.el9.x86_64 2024-04-06T02:39:35.585 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mds-2:18.2.1-0.el9.x86_64 2024-04-06T02:39:35.585 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-2:18.2.1-0.el9.x86_64 2024-04-06T02:39:35.585 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 2024-04-06T02:39:35.585 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mon-2:18.2.1-0.el9.x86_64 2024-04-06T02:39:35.585 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-osd-2:18.2.1-0.el9.x86_64 2024-04-06T02:39:35.585 INFO:teuthology.orchestra.run.smithi184.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-06T02:39:35.585 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-06T02:39:35.585 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-06T02:39:35.585 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-06T02:39:35.585 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-06T02:39:35.585 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-06T02:39:35.585 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-06T02:39:35.586 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-06T02:39:35.586 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-06T02:39:35.586 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-06T02:39:35.586 INFO:teuthology.orchestra.run.smithi184.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-06T02:39:35.586 INFO:teuthology.orchestra.run.smithi184.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-06T02:39:35.586 INFO:teuthology.orchestra.run.smithi184.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-06T02:39:35.586 INFO:teuthology.orchestra.run.smithi184.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-06T02:39:35.586 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-06T02:39:35.586 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ply-3.11-14.el9.noarch 2024-04-06T02:39:35.586 INFO:teuthology.orchestra.run.smithi184.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-06T02:39:35.586 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-06T02:39:35.586 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-06T02:39:35.586 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-06T02:39:35.586 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-06T02:39:35.586 INFO:teuthology.orchestra.run.smithi184.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-06T02:39:35.587 INFO:teuthology.orchestra.run.smithi184.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-06T02:39:35.587 INFO:teuthology.orchestra.run.smithi184.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-06T02:39:35.587 INFO:teuthology.orchestra.run.smithi184.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-06T02:39:35.587 INFO:teuthology.orchestra.run.smithi184.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-06T02:39:35.587 INFO:teuthology.orchestra.run.smithi184.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-06T02:39:35.587 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:35.587 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-04-06T02:39:35.604 INFO:teuthology.orchestra.run.smithi105.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 8.9 MB/s | 16 MB 00:01 2024-04-06T02:39:35.606 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:39:35.606 INFO:teuthology.orchestra.run.smithi105.stdout:Total 11 MB/s | 38 MB 00:03 2024-04-06T02:39:35.942 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-base 2024-04-06T02:39:35.961 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-06T02:39:35.987 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-06T02:39:35.987 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-06T02:39:36.497 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-06T02:39:36.497 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-06T02:39:36.533 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:58 ago on Sat 06 Apr 2024 02:38:38 AM UTC. 2024-04-06T02:39:36.821 INFO:teuthology.orchestra.run.smithi184.stderr:Modular dependency problems: 2024-04-06T02:39:36.822 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-04-06T02:39:36.822 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 1: conflicting requests 2024-04-06T02:39:36.822 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.822 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 2: conflicting requests 2024-04-06T02:39:36.822 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.822 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 3: conflicting requests 2024-04-06T02:39:36.822 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.822 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 4: conflicting requests 2024-04-06T02:39:36.822 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.822 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 5: conflicting requests 2024-04-06T02:39:36.822 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.823 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 6: conflicting requests 2024-04-06T02:39:36.823 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.823 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 7: conflicting requests 2024-04-06T02:39:36.823 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.823 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 8: conflicting requests 2024-04-06T02:39:36.823 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.823 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 9: conflicting requests 2024-04-06T02:39:36.823 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.823 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 10: conflicting requests 2024-04-06T02:39:36.823 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.823 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 11: conflicting requests 2024-04-06T02:39:36.823 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.823 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 12: conflicting requests 2024-04-06T02:39:36.823 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 13: conflicting requests 2024-04-06T02:39:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 14: conflicting requests 2024-04-06T02:39:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 15: conflicting requests 2024-04-06T02:39:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 16: conflicting requests 2024-04-06T02:39:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 17: conflicting requests 2024-04-06T02:39:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 18: conflicting requests 2024-04-06T02:39:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 19: conflicting requests 2024-04-06T02:39:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.825 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 20: conflicting requests 2024-04-06T02:39:36.825 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.825 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 21: conflicting requests 2024-04-06T02:39:36.825 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.825 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 22: conflicting requests 2024-04-06T02:39:36.825 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.825 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 23: conflicting requests 2024-04-06T02:39:36.825 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.825 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 24: conflicting requests 2024-04-06T02:39:36.825 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.825 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 25: conflicting requests 2024-04-06T02:39:36.825 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.825 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 26: conflicting requests 2024-04-06T02:39:36.826 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.826 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 27: conflicting requests 2024-04-06T02:39:36.826 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.826 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 28: conflicting requests 2024-04-06T02:39:36.826 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.826 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 29: conflicting requests 2024-04-06T02:39:36.826 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.826 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 30: conflicting requests 2024-04-06T02:39:36.826 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.826 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 31: conflicting requests 2024-04-06T02:39:36.826 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.826 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 32: conflicting requests 2024-04-06T02:39:36.826 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.827 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 33: conflicting requests 2024-04-06T02:39:36.827 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.827 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 34: conflicting requests 2024-04-06T02:39:36.827 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.827 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 35: conflicting requests 2024-04-06T02:39:36.827 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.827 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 36: conflicting requests 2024-04-06T02:39:36.827 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.827 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 37: conflicting requests 2024-04-06T02:39:36.827 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.827 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.827 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.827 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:39:36.827 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.828 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.828 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.828 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.828 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 39: conflicting requests 2024-04-06T02:39:36.828 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.828 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.828 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.828 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.828 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.828 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.828 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:39:36.828 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.828 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.828 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.829 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.829 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:39:36.829 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.829 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.829 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.829 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.829 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:39:36.829 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.829 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.829 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.829 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.829 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:39:36.829 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.829 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.830 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.830 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.830 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:39:36.830 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.830 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.830 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.830 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.830 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:39:36.830 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.830 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.830 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.830 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:39:36.831 INFO:teuthology.orchestra.run.smithi184.stdout:Package ceph-base-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-06T02:39:36.891 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-04-06T02:39:36.892 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-04-06T02:39:36.892 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-04-06T02:39:36.962 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install cephadm 2024-04-06T02:39:37.252 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-06T02:39:37.326 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-06T02:39:37.368 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-06T02:39:37.475 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-06T02:39:37.520 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-06T02:39:37.557 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:59 ago on Sat 06 Apr 2024 02:38:38 AM UTC. 2024-04-06T02:39:37.696 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-06T02:39:37.727 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-06T02:39:37.824 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-06T02:39:37.847 INFO:teuthology.orchestra.run.smithi184.stderr:Modular dependency problems: 2024-04-06T02:39:37.847 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-04-06T02:39:37.847 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 1: conflicting requests 2024-04-06T02:39:37.847 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.847 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 2: conflicting requests 2024-04-06T02:39:37.847 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.847 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 3: conflicting requests 2024-04-06T02:39:37.847 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.847 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 4: conflicting requests 2024-04-06T02:39:37.847 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.848 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 5: conflicting requests 2024-04-06T02:39:37.848 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.848 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 6: conflicting requests 2024-04-06T02:39:37.848 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.848 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 7: conflicting requests 2024-04-06T02:39:37.848 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.848 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 8: conflicting requests 2024-04-06T02:39:37.848 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.848 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 9: conflicting requests 2024-04-06T02:39:37.848 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.848 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 10: conflicting requests 2024-04-06T02:39:37.848 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.848 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 11: conflicting requests 2024-04-06T02:39:37.849 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.849 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 12: conflicting requests 2024-04-06T02:39:37.849 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.849 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 13: conflicting requests 2024-04-06T02:39:37.849 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.849 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 14: conflicting requests 2024-04-06T02:39:37.849 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.849 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 15: conflicting requests 2024-04-06T02:39:37.849 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.849 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 16: conflicting requests 2024-04-06T02:39:37.849 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.849 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 17: conflicting requests 2024-04-06T02:39:37.849 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.850 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 18: conflicting requests 2024-04-06T02:39:37.850 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.850 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 19: conflicting requests 2024-04-06T02:39:37.850 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.850 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 20: conflicting requests 2024-04-06T02:39:37.850 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.850 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 21: conflicting requests 2024-04-06T02:39:37.850 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.850 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 22: conflicting requests 2024-04-06T02:39:37.850 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.850 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 23: conflicting requests 2024-04-06T02:39:37.850 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.850 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 24: conflicting requests 2024-04-06T02:39:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 25: conflicting requests 2024-04-06T02:39:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 26: conflicting requests 2024-04-06T02:39:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 27: conflicting requests 2024-04-06T02:39:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 28: conflicting requests 2024-04-06T02:39:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 29: conflicting requests 2024-04-06T02:39:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 30: conflicting requests 2024-04-06T02:39:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 31: conflicting requests 2024-04-06T02:39:37.852 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.852 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 32: conflicting requests 2024-04-06T02:39:37.852 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.852 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 33: conflicting requests 2024-04-06T02:39:37.852 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.852 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 34: conflicting requests 2024-04-06T02:39:37.852 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.852 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 35: conflicting requests 2024-04-06T02:39:37.852 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.852 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 36: conflicting requests 2024-04-06T02:39:37.852 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.852 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 37: conflicting requests 2024-04-06T02:39:37.852 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.853 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.853 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.853 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:39:37.853 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.853 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.853 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.853 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.853 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 39: conflicting requests 2024-04-06T02:39:37.853 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.853 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.853 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.853 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.853 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.853 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.854 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:39:37.854 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.854 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.854 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.854 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.854 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:39:37.854 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.854 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.854 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.854 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.854 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:39:37.854 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.854 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.855 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.855 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.855 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:39:37.855 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.855 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.855 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.855 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.855 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:39:37.855 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.855 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.855 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.855 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.855 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:39:37.855 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.856 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.856 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.856 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:39:37.913 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-04-06T02:39:37.914 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:39:37.914 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-04-06T02:39:37.914 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:39:37.914 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-04-06T02:39:37.914 INFO:teuthology.orchestra.run.smithi184.stdout: cephadm noarch 2:18.2.1-0.el9 ceph-noarch 221 k 2024-04-06T02:39:37.914 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:37.914 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-04-06T02:39:37.915 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:39:37.915 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-04-06T02:39:37.915 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:37.915 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 221 k 2024-04-06T02:39:37.915 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 213 k 2024-04-06T02:39:37.915 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-04-06T02:39:37.991 INFO:teuthology.orchestra.run.smithi184.stdout:cephadm-18.2.1-0.el9.noarch.rpm 2.8 MB/s | 221 kB 00:00 2024-04-06T02:39:37.992 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:39:37.992 INFO:teuthology.orchestra.run.smithi184.stdout:Total 2.8 MB/s | 221 kB 00:00 2024-04-06T02:39:37.992 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-04-06T02:39:37.997 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-04-06T02:39:37.997 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-04-06T02:39:38.000 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-04-06T02:39:38.000 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-04-06T02:39:38.072 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-04-06T02:39:38.474 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-06T02:39:38.510 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-06T02:39:38.594 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-06T02:39:38.597 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-06T02:39:38.923 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-06T02:39:39.225 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-06T02:39:39.226 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:39.226 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-04-06T02:39:39.226 INFO:teuthology.orchestra.run.smithi184.stdout: cephadm-2:18.2.1-0.el9.noarch 2024-04-06T02:39:39.226 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:39.226 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-04-06T02:39:39.436 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-immutable-object-cache 2024-04-06T02:39:39.838 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-06T02:39:40.042 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:02 ago on Sat 06 Apr 2024 02:38:38 AM UTC. 2024-04-06T02:39:40.335 INFO:teuthology.orchestra.run.smithi184.stderr:Modular dependency problems: 2024-04-06T02:39:40.336 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-04-06T02:39:40.336 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 1: conflicting requests 2024-04-06T02:39:40.336 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.336 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 2: conflicting requests 2024-04-06T02:39:40.336 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.336 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 3: conflicting requests 2024-04-06T02:39:40.336 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.336 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 4: conflicting requests 2024-04-06T02:39:40.336 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.336 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 5: conflicting requests 2024-04-06T02:39:40.336 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.336 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 6: conflicting requests 2024-04-06T02:39:40.337 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.337 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 7: conflicting requests 2024-04-06T02:39:40.337 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.337 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 8: conflicting requests 2024-04-06T02:39:40.337 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.337 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 9: conflicting requests 2024-04-06T02:39:40.337 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.337 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 10: conflicting requests 2024-04-06T02:39:40.337 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.337 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 11: conflicting requests 2024-04-06T02:39:40.337 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.337 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 12: conflicting requests 2024-04-06T02:39:40.338 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.338 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 13: conflicting requests 2024-04-06T02:39:40.338 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.338 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 14: conflicting requests 2024-04-06T02:39:40.338 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.338 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 15: conflicting requests 2024-04-06T02:39:40.338 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.338 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 16: conflicting requests 2024-04-06T02:39:40.338 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.338 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 17: conflicting requests 2024-04-06T02:39:40.338 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.338 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 18: conflicting requests 2024-04-06T02:39:40.339 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.339 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 19: conflicting requests 2024-04-06T02:39:40.339 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.339 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 20: conflicting requests 2024-04-06T02:39:40.339 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.339 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 21: conflicting requests 2024-04-06T02:39:40.339 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.339 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 22: conflicting requests 2024-04-06T02:39:40.339 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.339 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 23: conflicting requests 2024-04-06T02:39:40.339 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.339 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 24: conflicting requests 2024-04-06T02:39:40.339 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.340 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 25: conflicting requests 2024-04-06T02:39:40.340 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.340 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 26: conflicting requests 2024-04-06T02:39:40.340 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.340 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 27: conflicting requests 2024-04-06T02:39:40.340 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.340 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 28: conflicting requests 2024-04-06T02:39:40.340 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.340 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 29: conflicting requests 2024-04-06T02:39:40.340 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.340 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 30: conflicting requests 2024-04-06T02:39:40.340 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.340 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 31: conflicting requests 2024-04-06T02:39:40.341 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.341 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 32: conflicting requests 2024-04-06T02:39:40.341 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.341 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 33: conflicting requests 2024-04-06T02:39:40.341 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.341 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 34: conflicting requests 2024-04-06T02:39:40.341 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.341 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 35: conflicting requests 2024-04-06T02:39:40.341 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.341 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 36: conflicting requests 2024-04-06T02:39:40.341 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.341 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 37: conflicting requests 2024-04-06T02:39:40.341 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.341 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.341 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.342 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:39:40.342 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.342 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.342 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.342 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.342 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 39: conflicting requests 2024-04-06T02:39:40.342 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.342 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.342 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.342 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.342 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.342 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.342 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:39:40.343 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.343 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.343 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.343 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.343 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:39:40.343 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.343 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.343 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.343 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.343 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:39:40.343 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.343 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.344 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.344 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.344 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:39:40.344 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.344 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.344 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.344 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.344 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:39:40.344 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.344 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.344 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.344 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.344 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:39:40.344 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.344 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.345 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.345 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:39:40.403 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-04-06T02:39:40.403 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:39:40.403 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2024-04-06T02:39:40.403 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:39:40.403 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-04-06T02:39:40.404 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-0.el9 ceph 142 k 2024-04-06T02:39:40.404 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:40.404 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-04-06T02:39:40.404 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:39:40.404 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-04-06T02:39:40.404 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:40.404 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 142 k 2024-04-06T02:39:40.405 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 395 k 2024-04-06T02:39:40.405 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-04-06T02:39:40.480 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-immutable-object-cache-18.2.1-0.el9.x86_64 1.8 MB/s | 142 kB 00:00 2024-04-06T02:39:40.481 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:39:40.481 INFO:teuthology.orchestra.run.smithi184.stdout:Total 1.8 MB/s | 142 kB 00:00 2024-04-06T02:39:40.481 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-04-06T02:39:40.489 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-04-06T02:39:40.489 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-04-06T02:39:40.542 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-04-06T02:39:40.542 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-04-06T02:39:40.680 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-04-06T02:39:40.739 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:39:40.764 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:39:40.765 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-06T02:39:40.765 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-06T02:39:40.765 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:40.851 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-06T02:39:40.885 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-04-06T02:39:41.801 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:39:41.801 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:41.801 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-04-06T02:39:41.801 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 2024-04-06T02:39:41.801 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:41.801 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-04-06T02:39:42.041 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr 2024-04-06T02:39:42.658 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:04 ago on Sat 06 Apr 2024 02:38:38 AM UTC. 2024-04-06T02:39:42.958 INFO:teuthology.orchestra.run.smithi184.stderr:Modular dependency problems: 2024-04-06T02:39:42.958 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-04-06T02:39:42.958 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 1: conflicting requests 2024-04-06T02:39:42.958 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.958 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 2: conflicting requests 2024-04-06T02:39:42.958 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.958 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 3: conflicting requests 2024-04-06T02:39:42.958 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.958 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 4: conflicting requests 2024-04-06T02:39:42.958 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.958 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 5: conflicting requests 2024-04-06T02:39:42.959 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.959 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 6: conflicting requests 2024-04-06T02:39:42.959 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.959 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 7: conflicting requests 2024-04-06T02:39:42.959 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.959 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 8: conflicting requests 2024-04-06T02:39:42.959 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.959 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 9: conflicting requests 2024-04-06T02:39:42.959 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.959 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 10: conflicting requests 2024-04-06T02:39:42.959 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.959 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 11: conflicting requests 2024-04-06T02:39:42.959 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.959 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 12: conflicting requests 2024-04-06T02:39:42.960 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.960 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 13: conflicting requests 2024-04-06T02:39:42.960 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.960 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 14: conflicting requests 2024-04-06T02:39:42.960 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.960 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 15: conflicting requests 2024-04-06T02:39:42.960 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.960 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 16: conflicting requests 2024-04-06T02:39:42.960 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.960 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 17: conflicting requests 2024-04-06T02:39:42.960 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.960 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 18: conflicting requests 2024-04-06T02:39:42.960 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.960 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 19: conflicting requests 2024-04-06T02:39:42.961 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.961 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 20: conflicting requests 2024-04-06T02:39:42.961 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.961 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 21: conflicting requests 2024-04-06T02:39:42.961 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.961 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 22: conflicting requests 2024-04-06T02:39:42.961 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.961 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 23: conflicting requests 2024-04-06T02:39:42.961 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.961 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 24: conflicting requests 2024-04-06T02:39:42.961 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.962 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 25: conflicting requests 2024-04-06T02:39:42.962 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.962 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 26: conflicting requests 2024-04-06T02:39:42.962 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.962 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 27: conflicting requests 2024-04-06T02:39:42.962 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.962 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 28: conflicting requests 2024-04-06T02:39:42.962 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.962 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 29: conflicting requests 2024-04-06T02:39:42.962 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.962 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 30: conflicting requests 2024-04-06T02:39:42.962 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.962 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 31: conflicting requests 2024-04-06T02:39:42.963 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.963 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 32: conflicting requests 2024-04-06T02:39:42.963 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.963 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 33: conflicting requests 2024-04-06T02:39:42.963 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.963 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 34: conflicting requests 2024-04-06T02:39:42.963 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.963 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 35: conflicting requests 2024-04-06T02:39:42.963 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.963 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 36: conflicting requests 2024-04-06T02:39:42.963 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.963 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 37: conflicting requests 2024-04-06T02:39:42.963 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.963 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.964 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.964 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:39:42.964 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.964 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.964 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.964 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.964 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 39: conflicting requests 2024-04-06T02:39:42.964 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.964 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.964 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.964 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.964 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.964 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.964 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:39:42.965 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.965 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.965 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.965 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.965 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:39:42.965 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.965 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.965 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.965 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.965 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:39:42.965 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.965 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.965 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.965 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.966 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:39:42.966 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.966 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.966 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.966 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.966 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:39:42.966 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.966 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.966 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.966 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.966 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:39:42.966 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.966 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.967 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.967 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:39:42.967 INFO:teuthology.orchestra.run.smithi184.stdout:Package ceph-mgr-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-06T02:39:43.033 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-04-06T02:39:43.034 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-04-06T02:39:43.034 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-04-06T02:39:43.105 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-dashboard 2024-04-06T02:39:43.391 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-04-06T02:39:43.392 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 1/12 2024-04-06T02:39:43.392 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-06T02:39:43.392 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-06T02:39:43.392 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-06T02:39:43.392 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-06T02:39:43.392 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-06T02:39:43.392 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-06T02:39:43.392 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-06T02:39:43.392 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 9/12 2024-04-06T02:39:43.392 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-06T02:39:43.392 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-06T02:39:43.695 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:05 ago on Sat 06 Apr 2024 02:38:38 AM UTC. 2024-04-06T02:39:43.865 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-06T02:39:43.865 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:43.866 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-06T02:39:43.866 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarch 2024-04-06T02:39:43.866 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-06T02:39:43.866 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-06T02:39:43.866 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-06T02:39:43.866 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-06T02:39:43.866 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-06T02:39:43.866 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-06T02:39:43.866 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-06T02:39:43.866 INFO:teuthology.orchestra.run.smithi105.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-06T02:39:43.866 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-06T02:39:43.866 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-06T02:39:43.866 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-06T02:39:43.866 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:43.867 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-06T02:39:43.983 INFO:teuthology.orchestra.run.smithi184.stderr:Modular dependency problems: 2024-04-06T02:39:43.983 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-04-06T02:39:43.983 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 1: conflicting requests 2024-04-06T02:39:43.983 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.983 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 2: conflicting requests 2024-04-06T02:39:43.983 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.984 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 3: conflicting requests 2024-04-06T02:39:43.984 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.984 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 4: conflicting requests 2024-04-06T02:39:43.984 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.984 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 5: conflicting requests 2024-04-06T02:39:43.984 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.984 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 6: conflicting requests 2024-04-06T02:39:43.984 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.984 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 7: conflicting requests 2024-04-06T02:39:43.984 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.984 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 8: conflicting requests 2024-04-06T02:39:43.984 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.984 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 9: conflicting requests 2024-04-06T02:39:43.984 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.985 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 10: conflicting requests 2024-04-06T02:39:43.985 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.985 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 11: conflicting requests 2024-04-06T02:39:43.985 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.985 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 12: conflicting requests 2024-04-06T02:39:43.985 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.985 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 13: conflicting requests 2024-04-06T02:39:43.985 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.985 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 14: conflicting requests 2024-04-06T02:39:43.985 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.985 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 15: conflicting requests 2024-04-06T02:39:43.985 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.985 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 16: conflicting requests 2024-04-06T02:39:43.986 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.986 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 17: conflicting requests 2024-04-06T02:39:43.986 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.986 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 18: conflicting requests 2024-04-06T02:39:43.986 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.986 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 19: conflicting requests 2024-04-06T02:39:43.986 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.986 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 20: conflicting requests 2024-04-06T02:39:43.986 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.986 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 21: conflicting requests 2024-04-06T02:39:43.986 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.986 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 22: conflicting requests 2024-04-06T02:39:43.986 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.986 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 23: conflicting requests 2024-04-06T02:39:43.987 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.987 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 24: conflicting requests 2024-04-06T02:39:43.987 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.987 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 25: conflicting requests 2024-04-06T02:39:43.987 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.987 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 26: conflicting requests 2024-04-06T02:39:43.987 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.987 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 27: conflicting requests 2024-04-06T02:39:43.987 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.987 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 28: conflicting requests 2024-04-06T02:39:43.987 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.987 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 29: conflicting requests 2024-04-06T02:39:43.987 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.987 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 30: conflicting requests 2024-04-06T02:39:43.988 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.988 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 31: conflicting requests 2024-04-06T02:39:43.988 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.988 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 32: conflicting requests 2024-04-06T02:39:43.988 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.988 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 33: conflicting requests 2024-04-06T02:39:43.988 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.988 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 34: conflicting requests 2024-04-06T02:39:43.988 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.988 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 35: conflicting requests 2024-04-06T02:39:43.988 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.988 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 36: conflicting requests 2024-04-06T02:39:43.988 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.988 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 37: conflicting requests 2024-04-06T02:39:43.989 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.989 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.989 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.989 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:39:43.989 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.989 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.989 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.989 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.989 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 39: conflicting requests 2024-04-06T02:39:43.989 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.989 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.989 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.989 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.989 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.990 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.990 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:39:43.990 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.990 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.990 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.990 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.990 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:39:43.990 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.990 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.990 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.990 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.990 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:39:43.990 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.990 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.991 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.991 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.991 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:39:43.991 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.991 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.991 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.991 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.991 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:39:43.991 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.991 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.991 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.991 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.991 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:39:43.992 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.992 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.992 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:39:43.992 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:39:44.051 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-04-06T02:39:44.052 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:39:44.052 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-04-06T02:39:44.053 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:39:44.053 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-04-06T02:39:44.053 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-dashboard noarch 2:18.2.1-0.el9 ceph-noarch 1.8 M 2024-04-06T02:39:44.053 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-04-06T02:39:44.053 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-grafana-dashboards noarch 2:18.2.1-0.el9 ceph-noarch 23 k 2024-04-06T02:39:44.053 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-prometheus-alerts noarch 2:18.2.1-0.el9 ceph-noarch 15 k 2024-04-06T02:39:44.053 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-04-06T02:39:44.053 INFO:teuthology.orchestra.run.smithi184.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-06T02:39:44.053 INFO:teuthology.orchestra.run.smithi184.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-06T02:39:44.053 INFO:teuthology.orchestra.run.smithi184.stdout:Installing weak dependencies: 2024-04-06T02:39:44.053 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-04-06T02:39:44.053 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:44.053 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-04-06T02:39:44.054 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:39:44.054 INFO:teuthology.orchestra.run.smithi184.stdout:Install 7 Packages 2024-04-06T02:39:44.054 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:44.054 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 2.1 M 2024-04-06T02:39:44.054 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 8.5 M 2024-04-06T02:39:44.054 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-04-06T02:39:44.277 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-rook 2024-04-06T02:39:44.309 INFO:teuthology.orchestra.run.smithi184.stdout:(1/7): ceph-grafana-dashboards-18.2.1-0.el9.noa 215 kB/s | 23 kB 00:00 2024-04-06T02:39:44.350 INFO:teuthology.orchestra.run.smithi184.stdout:(2/7): ceph-prometheus-alerts-18.2.1-0.el9.noar 99 kB/s | 15 kB 00:00 2024-04-06T02:39:44.392 INFO:teuthology.orchestra.run.smithi184.stdout:(3/7): ceph-mgr-dashboard-18.2.1-0.el9.noarch.r 9.2 MB/s | 1.8 MB 00:00 2024-04-06T02:39:44.543 INFO:teuthology.orchestra.run.smithi184.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 39 kB/s | 9.0 kB 00:00 2024-04-06T02:39:44.635 INFO:teuthology.orchestra.run.smithi184.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 144 kB/s | 41 kB 00:00 2024-04-06T02:39:44.660 INFO:teuthology.orchestra.run.smithi184.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 139 kB/s | 31 kB 00:00 2024-04-06T02:39:44.735 INFO:teuthology.orchestra.run.smithi184.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 988 kB/s | 188 kB 00:00 2024-04-06T02:39:44.739 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:39:44.739 INFO:teuthology.orchestra.run.smithi184.stdout:Total 3.0 MB/s | 2.1 MB 00:00 2024-04-06T02:39:44.740 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-04-06T02:39:44.747 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-04-06T02:39:44.747 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-04-06T02:39:44.812 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-04-06T02:39:44.813 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-04-06T02:39:44.875 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:18 ago on Sat 06 Apr 2024 02:38:26 AM UTC. 2024-04-06T02:39:45.134 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-04-06T02:39:45.159 INFO:teuthology.orchestra.run.smithi105.stderr:Modular dependency problems: 2024-04-06T02:39:45.160 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-06T02:39:45.160 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 1: conflicting requests 2024-04-06T02:39:45.160 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.160 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 2: conflicting requests 2024-04-06T02:39:45.160 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.160 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 3: conflicting requests 2024-04-06T02:39:45.160 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.160 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 4: conflicting requests 2024-04-06T02:39:45.160 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.160 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 5: conflicting requests 2024-04-06T02:39:45.160 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.160 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 6: conflicting requests 2024-04-06T02:39:45.160 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.160 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 7: conflicting requests 2024-04-06T02:39:45.161 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.161 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 8: conflicting requests 2024-04-06T02:39:45.161 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.161 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 9: conflicting requests 2024-04-06T02:39:45.161 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.161 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 10: conflicting requests 2024-04-06T02:39:45.161 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.161 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 11: conflicting requests 2024-04-06T02:39:45.161 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.161 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 12: conflicting requests 2024-04-06T02:39:45.161 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.161 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 13: conflicting requests 2024-04-06T02:39:45.162 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.162 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 14: conflicting requests 2024-04-06T02:39:45.162 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.162 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 15: conflicting requests 2024-04-06T02:39:45.162 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.162 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 16: conflicting requests 2024-04-06T02:39:45.162 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.162 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 17: conflicting requests 2024-04-06T02:39:45.162 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.162 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 18: conflicting requests 2024-04-06T02:39:45.162 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.162 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 19: conflicting requests 2024-04-06T02:39:45.163 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.163 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 20: conflicting requests 2024-04-06T02:39:45.163 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.163 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 21: conflicting requests 2024-04-06T02:39:45.163 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.163 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 22: conflicting requests 2024-04-06T02:39:45.163 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.163 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 23: conflicting requests 2024-04-06T02:39:45.163 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.163 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 24: conflicting requests 2024-04-06T02:39:45.163 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.163 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 25: conflicting requests 2024-04-06T02:39:45.163 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.163 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 26: conflicting requests 2024-04-06T02:39:45.164 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.164 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 27: conflicting requests 2024-04-06T02:39:45.164 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.164 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 28: conflicting requests 2024-04-06T02:39:45.164 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.164 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 29: conflicting requests 2024-04-06T02:39:45.164 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.164 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 30: conflicting requests 2024-04-06T02:39:45.164 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.164 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 31: conflicting requests 2024-04-06T02:39:45.164 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.165 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 32: conflicting requests 2024-04-06T02:39:45.165 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.165 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 33: conflicting requests 2024-04-06T02:39:45.165 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.165 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 34: conflicting requests 2024-04-06T02:39:45.165 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.165 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 35: conflicting requests 2024-04-06T02:39:45.165 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.165 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 36: conflicting requests 2024-04-06T02:39:45.165 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.165 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 37: conflicting requests 2024-04-06T02:39:45.165 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.165 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.165 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.166 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:39:45.166 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.166 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.166 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.166 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.166 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 39: conflicting requests 2024-04-06T02:39:45.166 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.166 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.166 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.166 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.166 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.166 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.167 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:39:45.167 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.167 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.167 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.167 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.167 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:39:45.167 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.167 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.167 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.167 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.167 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:39:45.167 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.167 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.167 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.167 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.168 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:39:45.168 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.168 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.168 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.168 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.168 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:39:45.168 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.168 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.168 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.168 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.168 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:39:45.168 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.168 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.168 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.169 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:39:45.195 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-04-06T02:39:45.227 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-06T02:39:45.228 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-04-06T02:39:45.229 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:39:45.229 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-04-06T02:39:45.229 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:39:45.229 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-06T02:39:45.229 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook noarch 2:18.2.1-0.el9 ceph-noarch 50 k 2024-04-06T02:39:45.229 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-04-06T02:39:45.229 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-06T02:39:45.229 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-06T02:39:45.229 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-06T02:39:45.229 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-06T02:39:45.229 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-06T02:39:45.229 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-06T02:39:45.229 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-06T02:39:45.230 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-06T02:39:45.230 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-06T02:39:45.230 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-06T02:39:45.230 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-06T02:39:45.230 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-06T02:39:45.230 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:45.230 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-06T02:39:45.230 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:39:45.230 INFO:teuthology.orchestra.run.smithi105.stdout:Install 13 Packages 2024-04-06T02:39:45.230 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:45.231 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 2.2 M 2024-04-06T02:39:45.231 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 26 M 2024-04-06T02:39:45.231 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-06T02:39:45.274 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-04-06T02:39:45.307 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-04-06T02:39:45.336 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 5/7 2024-04-06T02:39:45.510 INFO:teuthology.orchestra.run.smithi105.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 313 kB/s | 26 kB 00:00 2024-04-06T02:39:45.527 INFO:teuthology.orchestra.run.smithi105.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 195 kB/s | 19 kB 00:00 2024-04-06T02:39:45.544 INFO:teuthology.orchestra.run.smithi105.stdout:(3/13): ceph-mgr-rook-18.2.1-0.el9.noarch.rpm 434 kB/s | 50 kB 00:00 2024-04-06T02:39:45.571 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 6/7 2024-04-06T02:39:45.602 INFO:teuthology.orchestra.run.smithi105.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-04-06T02:39:45.605 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-04-06T02:39:45.628 INFO:teuthology.orchestra.run.smithi105.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 222 kB 00:00 2024-04-06T02:39:45.653 INFO:teuthology.orchestra.run.smithi105.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.5 MB/s | 279 kB 00:00 2024-04-06T02:39:45.687 INFO:teuthology.orchestra.run.smithi105.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 638 kB/s | 54 kB 00:00 2024-04-06T02:39:45.787 INFO:teuthology.orchestra.run.smithi105.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 112 kB/s | 15 kB 00:00 2024-04-06T02:39:45.813 INFO:teuthology.orchestra.run.smithi105.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 175 kB/s | 32 kB 00:00 2024-04-06T02:39:45.896 INFO:teuthology.orchestra.run.smithi105.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 970 kB/s | 203 kB 00:00 2024-04-06T02:39:45.921 INFO:teuthology.orchestra.run.smithi105.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 545 kB/s | 59 kB 00:00 2024-04-06T02:39:45.963 INFO:teuthology.orchestra.run.smithi105.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-04-06T02:39:46.038 INFO:teuthology.orchestra.run.smithi105.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.1 MB/s | 1.0 MB 00:00 2024-04-06T02:39:46.039 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:39:46.039 INFO:teuthology.orchestra.run.smithi105.stdout:Total 2.7 MB/s | 2.2 MB 00:00 2024-04-06T02:39:46.100 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-06T02:39:46.104 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-04-06T02:39:46.104 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 1/7 2024-04-06T02:39:46.104 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2/7 2024-04-06T02:39:46.104 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 3/7 2024-04-06T02:39:46.104 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-04-06T02:39:46.104 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-04-06T02:39:46.105 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-04-06T02:39:46.114 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-06T02:39:46.115 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-06T02:39:46.209 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-06T02:39:46.209 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-06T02:39:46.432 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-04-06T02:39:46.432 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:46.432 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-04-06T02:39:46.432 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 2024-04-06T02:39:46.433 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2024-04-06T02:39:46.433 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 2024-04-06T02:39:46.433 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-04-06T02:39:46.433 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-04-06T02:39:46.433 INFO:teuthology.orchestra.run.smithi184.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-06T02:39:46.433 INFO:teuthology.orchestra.run.smithi184.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-06T02:39:46.433 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:46.433 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-04-06T02:39:46.643 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-06T02:39:46.716 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-06T02:39:46.888 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-06T02:39:46.948 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-06T02:39:47.011 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-06T02:39:47.048 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-06T02:39:47.090 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-06T02:39:47.182 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-06T02:39:47.241 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:09 ago on Sat 06 Apr 2024 02:38:38 AM UTC. 2024-04-06T02:39:47.284 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-06T02:39:47.333 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-06T02:39:47.531 INFO:teuthology.orchestra.run.smithi184.stderr:Modular dependency problems: 2024-04-06T02:39:47.531 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-04-06T02:39:47.531 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 1: conflicting requests 2024-04-06T02:39:47.531 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.531 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 2: conflicting requests 2024-04-06T02:39:47.531 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.531 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 3: conflicting requests 2024-04-06T02:39:47.531 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.531 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 4: conflicting requests 2024-04-06T02:39:47.531 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.532 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 5: conflicting requests 2024-04-06T02:39:47.532 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.532 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 6: conflicting requests 2024-04-06T02:39:47.532 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.532 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 7: conflicting requests 2024-04-06T02:39:47.532 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.532 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 8: conflicting requests 2024-04-06T02:39:47.532 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.532 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 9: conflicting requests 2024-04-06T02:39:47.532 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.532 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 10: conflicting requests 2024-04-06T02:39:47.532 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.532 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 11: conflicting requests 2024-04-06T02:39:47.533 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.533 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 12: conflicting requests 2024-04-06T02:39:47.533 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.533 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 13: conflicting requests 2024-04-06T02:39:47.533 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.533 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 14: conflicting requests 2024-04-06T02:39:47.533 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.533 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 15: conflicting requests 2024-04-06T02:39:47.533 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.533 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 16: conflicting requests 2024-04-06T02:39:47.533 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.533 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 17: conflicting requests 2024-04-06T02:39:47.533 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.533 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 18: conflicting requests 2024-04-06T02:39:47.534 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.534 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 19: conflicting requests 2024-04-06T02:39:47.534 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.534 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 20: conflicting requests 2024-04-06T02:39:47.534 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.534 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 21: conflicting requests 2024-04-06T02:39:47.534 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.534 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 22: conflicting requests 2024-04-06T02:39:47.534 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.534 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 23: conflicting requests 2024-04-06T02:39:47.534 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.534 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 24: conflicting requests 2024-04-06T02:39:47.534 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.535 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 25: conflicting requests 2024-04-06T02:39:47.535 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.535 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 26: conflicting requests 2024-04-06T02:39:47.535 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.535 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 27: conflicting requests 2024-04-06T02:39:47.535 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.535 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 28: conflicting requests 2024-04-06T02:39:47.535 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.535 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 29: conflicting requests 2024-04-06T02:39:47.535 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.535 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 30: conflicting requests 2024-04-06T02:39:47.535 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.535 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 31: conflicting requests 2024-04-06T02:39:47.535 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.536 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 32: conflicting requests 2024-04-06T02:39:47.536 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.536 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 33: conflicting requests 2024-04-06T02:39:47.536 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.536 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 34: conflicting requests 2024-04-06T02:39:47.536 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.536 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 35: conflicting requests 2024-04-06T02:39:47.536 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.536 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 36: conflicting requests 2024-04-06T02:39:47.536 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.536 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 37: conflicting requests 2024-04-06T02:39:47.536 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.536 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.536 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.537 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:39:47.537 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.537 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.537 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.537 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.537 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 39: conflicting requests 2024-04-06T02:39:47.537 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.537 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.537 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.537 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.537 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.537 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.537 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:39:47.538 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.538 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.538 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.538 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.538 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:39:47.538 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.538 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.538 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.538 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.538 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:39:47.538 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.538 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.538 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.539 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.539 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:39:47.539 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.539 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.539 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.539 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.539 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:39:47.539 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.539 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.539 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.539 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.539 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:39:47.539 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.539 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.540 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.540 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:39:47.601 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-04-06T02:39:47.602 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:39:47.602 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-04-06T02:39:47.602 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:39:47.602 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-04-06T02:39:47.602 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-0.el9 ceph-noarch 7.4 M 2024-04-06T02:39:47.602 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-04-06T02:39:47.602 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-06T02:39:47.602 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-06T02:39:47.602 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-06T02:39:47.602 INFO:teuthology.orchestra.run.smithi184.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-06T02:39:47.603 INFO:teuthology.orchestra.run.smithi184.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-06T02:39:47.603 INFO:teuthology.orchestra.run.smithi184.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-06T02:39:47.603 INFO:teuthology.orchestra.run.smithi184.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-06T02:39:47.603 INFO:teuthology.orchestra.run.smithi184.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-06T02:39:47.603 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-06T02:39:47.603 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-06T02:39:47.603 INFO:teuthology.orchestra.run.smithi184.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-06T02:39:47.603 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:47.603 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-04-06T02:39:47.603 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:39:47.603 INFO:teuthology.orchestra.run.smithi184.stdout:Install 12 Packages 2024-04-06T02:39:47.603 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:39:47.604 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 38 M 2024-04-06T02:39:47.604 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 209 M 2024-04-06T02:39:47.604 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-04-06T02:39:48.056 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-06T02:39:48.106 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-06T02:39:48.139 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-06T02:39:48.158 INFO:teuthology.orchestra.run.smithi184.stdout:(1/12): ceph-mgr-diskprediction-local-18.2.1-0. 50 MB/s | 7.4 MB 00:00 2024-04-06T02:39:48.187 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-06T02:39:48.209 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-04-06T02:39:48.275 INFO:teuthology.orchestra.run.smithi184.stdout:(2/12): libquadmath-11.4.1-3.el9.x86_64.rpm 737 kB/s | 195 kB 00:00 2024-04-06T02:39:48.417 INFO:teuthology.orchestra.run.smithi184.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 126 kB/s | 33 kB 00:00 2024-04-06T02:39:48.459 INFO:teuthology.orchestra.run.smithi184.stdout:(4/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-04-06T02:39:48.509 INFO:teuthology.orchestra.run.smithi184.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 198 kB/s | 18 kB 00:00 2024-04-06T02:39:48.718 INFO:teuthology.orchestra.run.smithi184.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 136 kB/s | 35 kB 00:00 2024-04-06T02:39:49.054 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-04-06T02:39:49.054 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-rook-2:18.2.1-0.el9.noarch 1/13 2024-04-06T02:39:49.054 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-06T02:39:49.054 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-06T02:39:49.054 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-06T02:39:49.054 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-06T02:39:49.054 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-06T02:39:49.054 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-06T02:39:49.054 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-06T02:39:49.054 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-06T02:39:49.055 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-06T02:39:49.055 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-06T02:39:49.055 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-06T02:39:49.078 INFO:teuthology.orchestra.run.smithi184.stdout:(7/12): python3-devel-3.9.17-2.el9.x86_64.rpm 684 kB/s | 245 kB 00:00 2024-04-06T02:39:49.251 INFO:teuthology.orchestra.run.smithi184.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.1 MB/s | 3.0 MB 00:00 2024-04-06T02:39:49.368 INFO:teuthology.orchestra.run.smithi184.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 250 kB 00:00 2024-04-06T02:39:49.431 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-06T02:39:49.431 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:49.431 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-06T02:39:49.431 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook-2:18.2.1-0.el9.noarch 2024-04-06T02:39:49.431 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-06T02:39:49.431 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-06T02:39:49.431 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-06T02:39:49.431 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-06T02:39:49.431 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-06T02:39:49.431 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-06T02:39:49.432 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-06T02:39:49.432 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-06T02:39:49.432 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-06T02:39:49.432 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-06T02:39:49.432 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-06T02:39:49.432 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-06T02:39:49.432 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:49.432 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-06T02:39:49.535 INFO:teuthology.orchestra.run.smithi184.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 4.6 MB/s | 4.7 MB 00:01 2024-04-06T02:39:49.721 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-cephadm 2024-04-06T02:39:49.952 INFO:teuthology.orchestra.run.smithi184.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.8 MB/s | 5.1 MB 00:00 2024-04-06T02:39:50.312 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:24 ago on Sat 06 Apr 2024 02:38:26 AM UTC. 2024-04-06T02:39:50.598 INFO:teuthology.orchestra.run.smithi105.stderr:Modular dependency problems: 2024-04-06T02:39:50.598 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-06T02:39:50.598 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 1: conflicting requests 2024-04-06T02:39:50.598 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.598 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 2: conflicting requests 2024-04-06T02:39:50.598 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.598 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 3: conflicting requests 2024-04-06T02:39:50.599 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.599 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 4: conflicting requests 2024-04-06T02:39:50.599 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.599 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 5: conflicting requests 2024-04-06T02:39:50.599 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.599 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 6: conflicting requests 2024-04-06T02:39:50.599 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.599 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 7: conflicting requests 2024-04-06T02:39:50.599 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.599 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 8: conflicting requests 2024-04-06T02:39:50.599 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.599 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 9: conflicting requests 2024-04-06T02:39:50.600 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.600 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 10: conflicting requests 2024-04-06T02:39:50.600 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.600 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 11: conflicting requests 2024-04-06T02:39:50.600 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.600 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 12: conflicting requests 2024-04-06T02:39:50.600 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.600 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 13: conflicting requests 2024-04-06T02:39:50.600 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.600 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 14: conflicting requests 2024-04-06T02:39:50.600 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.600 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 15: conflicting requests 2024-04-06T02:39:50.600 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.601 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 16: conflicting requests 2024-04-06T02:39:50.601 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.601 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 17: conflicting requests 2024-04-06T02:39:50.601 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.601 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 18: conflicting requests 2024-04-06T02:39:50.601 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.601 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 19: conflicting requests 2024-04-06T02:39:50.601 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.601 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 20: conflicting requests 2024-04-06T02:39:50.601 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.601 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 21: conflicting requests 2024-04-06T02:39:50.601 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.601 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 22: conflicting requests 2024-04-06T02:39:50.602 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.602 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 23: conflicting requests 2024-04-06T02:39:50.602 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.602 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 24: conflicting requests 2024-04-06T02:39:50.602 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.602 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 25: conflicting requests 2024-04-06T02:39:50.602 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.602 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 26: conflicting requests 2024-04-06T02:39:50.602 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.602 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 27: conflicting requests 2024-04-06T02:39:50.602 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.602 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 28: conflicting requests 2024-04-06T02:39:50.602 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.603 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 29: conflicting requests 2024-04-06T02:39:50.603 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.603 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 30: conflicting requests 2024-04-06T02:39:50.603 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.603 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 31: conflicting requests 2024-04-06T02:39:50.603 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.603 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 32: conflicting requests 2024-04-06T02:39:50.603 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.603 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 33: conflicting requests 2024-04-06T02:39:50.603 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.603 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 34: conflicting requests 2024-04-06T02:39:50.603 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.603 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 35: conflicting requests 2024-04-06T02:39:50.603 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.604 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 36: conflicting requests 2024-04-06T02:39:50.604 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.604 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 37: conflicting requests 2024-04-06T02:39:50.604 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.604 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.604 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.604 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:39:50.604 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.604 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.604 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.604 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.604 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 39: conflicting requests 2024-04-06T02:39:50.604 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.605 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.605 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.605 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.605 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.605 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.605 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:39:50.605 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.605 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.605 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.605 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.605 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:39:50.605 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.605 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.605 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.606 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.606 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:39:50.606 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.606 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.606 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.606 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.606 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:39:50.606 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.606 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.606 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.606 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.606 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:39:50.606 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.607 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.607 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.607 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.607 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:39:50.607 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.607 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.607 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.607 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:39:50.667 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-06T02:39:50.668 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:39:50.668 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-04-06T02:39:50.668 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:39:50.668 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-06T02:39:50.668 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm noarch 2:18.2.1-0.el9 ceph-noarch 132 k 2024-04-06T02:39:50.668 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-04-06T02:39:50.668 INFO:teuthology.orchestra.run.smithi105.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-06T02:39:50.668 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-06T02:39:50.668 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-06T02:39:50.668 INFO:teuthology.orchestra.run.smithi105.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-06T02:39:50.668 INFO:teuthology.orchestra.run.smithi105.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-06T02:39:50.669 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:50.669 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-06T02:39:50.669 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:39:50.669 INFO:teuthology.orchestra.run.smithi105.stdout:Install 6 Packages 2024-04-06T02:39:50.669 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:50.669 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 7.0 M 2024-04-06T02:39:50.669 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 33 M 2024-04-06T02:39:50.670 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-06T02:39:51.174 INFO:teuthology.orchestra.run.smithi105.stdout:(1/6): ceph-mgr-cephadm-18.2.1-0.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-04-06T02:39:51.275 INFO:teuthology.orchestra.run.smithi105.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-04-06T02:39:51.433 INFO:teuthology.orchestra.run.smithi105.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-04-06T02:39:51.909 INFO:teuthology.orchestra.run.smithi105.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 95 kB/s | 58 kB 00:00 2024-04-06T02:39:51.984 INFO:teuthology.orchestra.run.smithi105.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 97 kB/s | 51 kB 00:00 2024-04-06T02:39:52.417 INFO:teuthology.orchestra.run.smithi105.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 441 kB/s | 548 kB 00:01 2024-04-06T02:39:52.420 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:39:52.420 INFO:teuthology.orchestra.run.smithi105.stdout:Total 4.0 MB/s | 7.0 MB 00:01 2024-04-06T02:39:52.490 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-06T02:39:52.501 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-06T02:39:52.501 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-06T02:39:52.601 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-06T02:39:52.602 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-06T02:39:52.915 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-06T02:39:53.082 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-06T02:39:53.135 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-06T02:39:53.644 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-06T02:39:53.722 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-06T02:39:53.782 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-06T02:39:53.806 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-04-06T02:39:54.633 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-04-06T02:39:54.633 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 1/6 2024-04-06T02:39:54.633 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-06T02:39:54.633 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-06T02:39:54.633 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-06T02:39:54.633 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-06T02:39:54.947 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-06T02:39:54.989 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:54.990 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-06T02:39:54.990 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 2024-04-06T02:39:54.990 INFO:teuthology.orchestra.run.smithi105.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-06T02:39:54.990 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-06T02:39:54.990 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-06T02:39:54.990 INFO:teuthology.orchestra.run.smithi105.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-06T02:39:54.990 INFO:teuthology.orchestra.run.smithi105.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-06T02:39:54.990 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:54.990 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-06T02:39:55.196 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-fuse 2024-04-06T02:39:55.792 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:29 ago on Sat 06 Apr 2024 02:38:26 AM UTC. 2024-04-06T02:39:56.081 INFO:teuthology.orchestra.run.smithi105.stderr:Modular dependency problems: 2024-04-06T02:39:56.081 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-06T02:39:56.081 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 1: conflicting requests 2024-04-06T02:39:56.081 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.082 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 2: conflicting requests 2024-04-06T02:39:56.082 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.082 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 3: conflicting requests 2024-04-06T02:39:56.082 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.082 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 4: conflicting requests 2024-04-06T02:39:56.082 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.083 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 5: conflicting requests 2024-04-06T02:39:56.083 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.083 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 6: conflicting requests 2024-04-06T02:39:56.083 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.083 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 7: conflicting requests 2024-04-06T02:39:56.083 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.083 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 8: conflicting requests 2024-04-06T02:39:56.083 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.083 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 9: conflicting requests 2024-04-06T02:39:56.083 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.083 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 10: conflicting requests 2024-04-06T02:39:56.083 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.083 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 11: conflicting requests 2024-04-06T02:39:56.084 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.084 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 12: conflicting requests 2024-04-06T02:39:56.084 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.084 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 13: conflicting requests 2024-04-06T02:39:56.084 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.084 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 14: conflicting requests 2024-04-06T02:39:56.084 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.084 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 15: conflicting requests 2024-04-06T02:39:56.084 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.084 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 16: conflicting requests 2024-04-06T02:39:56.084 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.084 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 17: conflicting requests 2024-04-06T02:39:56.084 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.084 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 18: conflicting requests 2024-04-06T02:39:56.085 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.085 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 19: conflicting requests 2024-04-06T02:39:56.085 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.085 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 20: conflicting requests 2024-04-06T02:39:56.085 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.085 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 21: conflicting requests 2024-04-06T02:39:56.085 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.085 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 22: conflicting requests 2024-04-06T02:39:56.085 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.085 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 23: conflicting requests 2024-04-06T02:39:56.085 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.085 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 24: conflicting requests 2024-04-06T02:39:56.085 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.086 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 25: conflicting requests 2024-04-06T02:39:56.086 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.086 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 26: conflicting requests 2024-04-06T02:39:56.086 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.086 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 27: conflicting requests 2024-04-06T02:39:56.086 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.086 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 28: conflicting requests 2024-04-06T02:39:56.086 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.086 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 29: conflicting requests 2024-04-06T02:39:56.087 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.087 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 30: conflicting requests 2024-04-06T02:39:56.087 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.087 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 31: conflicting requests 2024-04-06T02:39:56.087 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.087 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 32: conflicting requests 2024-04-06T02:39:56.087 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.087 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 33: conflicting requests 2024-04-06T02:39:56.087 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.087 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 34: conflicting requests 2024-04-06T02:39:56.087 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.087 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 35: conflicting requests 2024-04-06T02:39:56.087 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.087 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 36: conflicting requests 2024-04-06T02:39:56.088 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.088 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 37: conflicting requests 2024-04-06T02:39:56.088 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.088 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.088 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.088 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:39:56.088 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.088 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.088 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.088 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.088 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 39: conflicting requests 2024-04-06T02:39:56.088 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.089 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.089 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.089 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.089 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.089 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.089 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:39:56.089 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.089 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.089 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.089 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.089 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:39:56.089 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.089 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.089 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.090 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.090 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:39:56.090 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.090 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.090 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.090 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.090 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:39:56.090 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.090 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.090 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.090 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.090 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:39:56.090 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.091 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.091 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.091 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.091 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:39:56.091 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.091 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.091 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.091 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:39:56.150 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-06T02:39:56.150 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:39:56.150 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2024-04-06T02:39:56.150 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:39:56.150 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-06T02:39:56.151 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse x86_64 2:18.2.1-0.el9 ceph 839 k 2024-04-06T02:39:56.151 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-04-06T02:39:56.151 INFO:teuthology.orchestra.run.smithi105.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-06T02:39:56.151 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:56.151 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-06T02:39:56.151 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:39:56.151 INFO:teuthology.orchestra.run.smithi105.stdout:Install 2 Packages 2024-04-06T02:39:56.151 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:56.152 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 920 k 2024-04-06T02:39:56.152 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 2.7 M 2024-04-06T02:39:56.152 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-06T02:39:56.377 INFO:teuthology.orchestra.run.smithi105.stdout:(1/2): ceph-fuse-18.2.1-0.el9.x86_64.rpm 7.2 MB/s | 839 kB 00:00 2024-04-06T02:39:56.402 INFO:teuthology.orchestra.run.smithi105.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 580 kB/s | 80 kB 00:00 2024-04-06T02:39:56.403 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:39:56.403 INFO:teuthology.orchestra.run.smithi105.stdout:Total 3.6 MB/s | 920 kB 00:00 2024-04-06T02:39:56.415 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-06T02:39:56.437 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-06T02:39:56.438 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-06T02:39:56.514 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-06T02:39:56.515 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-06T02:39:56.706 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-06T02:39:56.841 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-06T02:39:56.918 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-04-06T02:39:57.659 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-04-06T02:39:57.660 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-fuse-2:18.2.1-0.el9.x86_64 1/2 2024-04-06T02:39:57.977 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-06T02:39:57.977 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:57.977 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-06T02:39:57.977 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse-2:18.2.1-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-06T02:39:57.977 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:57.977 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-06T02:39:58.281 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install librados-devel 2024-04-06T02:39:58.882 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:32 ago on Sat 06 Apr 2024 02:38:26 AM UTC. 2024-04-06T02:39:59.171 INFO:teuthology.orchestra.run.smithi105.stderr:Modular dependency problems: 2024-04-06T02:39:59.171 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-06T02:39:59.171 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 1: conflicting requests 2024-04-06T02:39:59.171 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.171 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 2: conflicting requests 2024-04-06T02:39:59.171 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.171 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 3: conflicting requests 2024-04-06T02:39:59.171 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.171 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 4: conflicting requests 2024-04-06T02:39:59.171 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.172 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 5: conflicting requests 2024-04-06T02:39:59.172 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.172 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 6: conflicting requests 2024-04-06T02:39:59.172 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.172 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 7: conflicting requests 2024-04-06T02:39:59.172 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.172 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 8: conflicting requests 2024-04-06T02:39:59.172 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.172 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 9: conflicting requests 2024-04-06T02:39:59.172 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.172 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 10: conflicting requests 2024-04-06T02:39:59.172 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.172 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 11: conflicting requests 2024-04-06T02:39:59.172 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.172 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 12: conflicting requests 2024-04-06T02:39:59.173 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.173 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 13: conflicting requests 2024-04-06T02:39:59.173 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.173 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 14: conflicting requests 2024-04-06T02:39:59.173 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.173 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 15: conflicting requests 2024-04-06T02:39:59.173 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.173 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 16: conflicting requests 2024-04-06T02:39:59.173 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.173 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 17: conflicting requests 2024-04-06T02:39:59.173 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.173 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 18: conflicting requests 2024-04-06T02:39:59.173 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.173 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 19: conflicting requests 2024-04-06T02:39:59.174 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.174 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 20: conflicting requests 2024-04-06T02:39:59.174 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.174 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 21: conflicting requests 2024-04-06T02:39:59.174 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.174 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 22: conflicting requests 2024-04-06T02:39:59.174 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.174 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 23: conflicting requests 2024-04-06T02:39:59.174 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.174 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 24: conflicting requests 2024-04-06T02:39:59.174 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.174 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 25: conflicting requests 2024-04-06T02:39:59.174 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.175 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 26: conflicting requests 2024-04-06T02:39:59.175 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.175 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 27: conflicting requests 2024-04-06T02:39:59.175 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.175 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 28: conflicting requests 2024-04-06T02:39:59.175 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.175 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 29: conflicting requests 2024-04-06T02:39:59.175 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.175 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 30: conflicting requests 2024-04-06T02:39:59.175 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.175 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 31: conflicting requests 2024-04-06T02:39:59.175 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.175 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 32: conflicting requests 2024-04-06T02:39:59.175 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.175 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 33: conflicting requests 2024-04-06T02:39:59.176 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.176 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 34: conflicting requests 2024-04-06T02:39:59.176 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.176 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 35: conflicting requests 2024-04-06T02:39:59.176 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.176 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 36: conflicting requests 2024-04-06T02:39:59.176 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.176 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 37: conflicting requests 2024-04-06T02:39:59.176 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.176 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.176 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.176 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:39:59.176 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.176 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.176 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.177 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.177 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 39: conflicting requests 2024-04-06T02:39:59.177 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.177 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.177 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.177 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.177 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.177 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.177 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:39:59.177 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.177 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.177 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.177 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.177 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:39:59.178 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.178 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.178 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.178 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.178 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:39:59.178 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.178 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.178 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.178 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.178 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:39:59.179 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.179 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.179 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.179 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.179 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:39:59.179 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.179 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.179 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.179 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.179 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:39:59.179 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.179 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.179 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.179 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:39:59.242 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-06T02:39:59.243 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:39:59.243 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2024-04-06T02:39:59.243 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:39:59.243 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-06T02:39:59.243 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel x86_64 2:18.2.1-0.el9 ceph 127 k 2024-04-06T02:39:59.243 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:59.243 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-06T02:39:59.243 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:39:59.243 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-04-06T02:39:59.243 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:39:59.244 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 127 k 2024-04-06T02:39:59.244 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 456 k 2024-04-06T02:39:59.244 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-06T02:39:59.332 INFO:teuthology.orchestra.run.smithi105.stdout:librados-devel-18.2.1-0.el9.x86_64.rpm 1.4 MB/s | 127 kB 00:00 2024-04-06T02:39:59.333 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:39:59.333 INFO:teuthology.orchestra.run.smithi105.stdout:Total 1.4 MB/s | 127 kB 00:00 2024-04-06T02:39:59.333 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-06T02:39:59.339 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-06T02:39:59.339 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-06T02:39:59.397 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-06T02:39:59.398 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-06T02:39:59.550 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-06T02:39:59.647 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:00.099 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:00.383 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:00.383 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:40:00.383 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-06T02:40:00.383 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel-2:18.2.1-0.el9.x86_64 2024-04-06T02:40:00.383 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:40:00.383 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-06T02:40:00.667 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install libcephfs2 2024-04-06T02:40:01.277 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:35 ago on Sat 06 Apr 2024 02:38:26 AM UTC. 2024-04-06T02:40:01.571 INFO:teuthology.orchestra.run.smithi105.stderr:Modular dependency problems: 2024-04-06T02:40:01.572 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-06T02:40:01.572 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 1: conflicting requests 2024-04-06T02:40:01.572 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.572 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 2: conflicting requests 2024-04-06T02:40:01.572 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.572 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 3: conflicting requests 2024-04-06T02:40:01.572 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.572 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 4: conflicting requests 2024-04-06T02:40:01.572 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.572 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 5: conflicting requests 2024-04-06T02:40:01.572 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.572 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 6: conflicting requests 2024-04-06T02:40:01.573 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.573 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 7: conflicting requests 2024-04-06T02:40:01.573 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.573 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 8: conflicting requests 2024-04-06T02:40:01.573 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.573 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 9: conflicting requests 2024-04-06T02:40:01.573 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.573 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 10: conflicting requests 2024-04-06T02:40:01.573 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.573 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 11: conflicting requests 2024-04-06T02:40:01.573 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.573 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 12: conflicting requests 2024-04-06T02:40:01.573 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.573 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 13: conflicting requests 2024-04-06T02:40:01.574 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.574 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 14: conflicting requests 2024-04-06T02:40:01.574 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.574 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 15: conflicting requests 2024-04-06T02:40:01.574 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.574 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 16: conflicting requests 2024-04-06T02:40:01.574 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.574 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 17: conflicting requests 2024-04-06T02:40:01.574 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.574 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 18: conflicting requests 2024-04-06T02:40:01.574 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.574 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 19: conflicting requests 2024-04-06T02:40:01.574 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.575 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 20: conflicting requests 2024-04-06T02:40:01.575 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.575 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 21: conflicting requests 2024-04-06T02:40:01.575 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.575 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 22: conflicting requests 2024-04-06T02:40:01.575 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.575 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 23: conflicting requests 2024-04-06T02:40:01.575 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.575 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 24: conflicting requests 2024-04-06T02:40:01.575 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.575 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 25: conflicting requests 2024-04-06T02:40:01.575 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.575 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 26: conflicting requests 2024-04-06T02:40:01.576 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.576 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 27: conflicting requests 2024-04-06T02:40:01.576 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.576 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 28: conflicting requests 2024-04-06T02:40:01.576 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.576 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 29: conflicting requests 2024-04-06T02:40:01.576 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.576 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 30: conflicting requests 2024-04-06T02:40:01.576 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.576 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 31: conflicting requests 2024-04-06T02:40:01.576 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.576 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 32: conflicting requests 2024-04-06T02:40:01.576 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.576 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 33: conflicting requests 2024-04-06T02:40:01.577 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.577 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 34: conflicting requests 2024-04-06T02:40:01.577 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.577 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 35: conflicting requests 2024-04-06T02:40:01.577 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.577 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 36: conflicting requests 2024-04-06T02:40:01.577 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.577 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 37: conflicting requests 2024-04-06T02:40:01.577 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.577 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.577 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.577 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:40:01.577 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.577 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.578 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.578 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.578 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 39: conflicting requests 2024-04-06T02:40:01.578 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.578 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.578 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.578 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.578 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.578 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.578 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:40:01.578 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.579 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.579 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.579 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.579 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:40:01.579 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.579 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.579 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.579 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.579 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:40:01.579 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.579 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.580 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.580 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.580 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:40:01.580 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.580 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.580 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.580 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.580 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:40:01.580 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.580 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.580 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.580 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.580 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:40:01.580 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.581 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.581 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.581 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:40:01.581 INFO:teuthology.orchestra.run.smithi105.stdout:Package libcephfs2-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-06T02:40:01.642 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-06T02:40:01.643 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-04-06T02:40:01.643 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-06T02:40:01.715 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install libcephfs-devel 2024-04-06T02:40:02.309 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:36 ago on Sat 06 Apr 2024 02:38:26 AM UTC. 2024-04-06T02:40:02.596 INFO:teuthology.orchestra.run.smithi105.stderr:Modular dependency problems: 2024-04-06T02:40:02.596 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-06T02:40:02.596 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 1: conflicting requests 2024-04-06T02:40:02.596 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.596 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 2: conflicting requests 2024-04-06T02:40:02.596 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.596 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 3: conflicting requests 2024-04-06T02:40:02.596 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.596 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 4: conflicting requests 2024-04-06T02:40:02.596 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.596 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 5: conflicting requests 2024-04-06T02:40:02.597 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.597 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 6: conflicting requests 2024-04-06T02:40:02.597 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.597 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 7: conflicting requests 2024-04-06T02:40:02.597 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.597 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 8: conflicting requests 2024-04-06T02:40:02.597 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.597 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 9: conflicting requests 2024-04-06T02:40:02.597 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.597 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 10: conflicting requests 2024-04-06T02:40:02.597 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.597 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 11: conflicting requests 2024-04-06T02:40:02.597 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.597 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 12: conflicting requests 2024-04-06T02:40:02.598 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.598 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 13: conflicting requests 2024-04-06T02:40:02.598 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.598 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 14: conflicting requests 2024-04-06T02:40:02.598 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.598 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 15: conflicting requests 2024-04-06T02:40:02.598 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.598 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 16: conflicting requests 2024-04-06T02:40:02.598 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.598 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 17: conflicting requests 2024-04-06T02:40:02.598 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.598 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 18: conflicting requests 2024-04-06T02:40:02.599 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.599 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 19: conflicting requests 2024-04-06T02:40:02.599 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.599 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 20: conflicting requests 2024-04-06T02:40:02.599 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.599 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 21: conflicting requests 2024-04-06T02:40:02.599 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.599 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 22: conflicting requests 2024-04-06T02:40:02.599 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.599 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 23: conflicting requests 2024-04-06T02:40:02.599 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.599 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 24: conflicting requests 2024-04-06T02:40:02.599 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.600 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 25: conflicting requests 2024-04-06T02:40:02.600 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.600 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 26: conflicting requests 2024-04-06T02:40:02.600 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.600 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 27: conflicting requests 2024-04-06T02:40:02.600 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.600 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 28: conflicting requests 2024-04-06T02:40:02.600 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.600 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 29: conflicting requests 2024-04-06T02:40:02.600 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.600 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 30: conflicting requests 2024-04-06T02:40:02.601 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.601 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 31: conflicting requests 2024-04-06T02:40:02.601 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.601 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 32: conflicting requests 2024-04-06T02:40:02.601 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.601 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 33: conflicting requests 2024-04-06T02:40:02.601 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.601 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 34: conflicting requests 2024-04-06T02:40:02.601 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.601 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 35: conflicting requests 2024-04-06T02:40:02.601 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.601 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 36: conflicting requests 2024-04-06T02:40:02.601 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.601 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 37: conflicting requests 2024-04-06T02:40:02.602 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.602 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.602 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.602 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:40:02.602 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.602 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.602 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.602 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.602 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 39: conflicting requests 2024-04-06T02:40:02.602 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.602 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.603 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.603 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.603 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.603 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.603 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:40:02.603 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.603 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.603 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.603 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.603 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:40:02.603 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.603 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.603 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.603 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.604 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:40:02.604 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.604 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.604 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.604 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.604 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:40:02.604 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.604 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.604 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.604 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.604 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:40:02.604 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.604 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.605 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.605 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.605 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:40:02.605 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.605 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.605 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.605 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:40:02.664 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-06T02:40:02.664 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:40:02.664 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2024-04-06T02:40:02.664 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:40:02.664 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-06T02:40:02.665 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel x86_64 2:18.2.1-0.el9 ceph 31 k 2024-04-06T02:40:02.665 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:40:02.665 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-06T02:40:02.665 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:40:02.665 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-04-06T02:40:02.665 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:40:02.665 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 31 k 2024-04-06T02:40:02.666 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 139 k 2024-04-06T02:40:02.666 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-06T02:40:02.754 INFO:teuthology.orchestra.run.smithi105.stdout:libcephfs-devel-18.2.1-0.el9.x86_64.rpm 349 kB/s | 31 kB 00:00 2024-04-06T02:40:02.755 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:40:02.755 INFO:teuthology.orchestra.run.smithi105.stdout:Total 344 kB/s | 31 kB 00:00 2024-04-06T02:40:02.755 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-06T02:40:02.758 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-06T02:40:02.759 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-06T02:40:02.790 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-06T02:40:02.790 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-06T02:40:02.873 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-06T02:40:02.969 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:03.339 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:03.619 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:03.619 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:40:03.620 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-06T02:40:03.620 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel-2:18.2.1-0.el9.x86_64 2024-04-06T02:40:03.620 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:40:03.620 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-06T02:40:03.818 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install librados2 2024-04-06T02:40:04.408 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:38 ago on Sat 06 Apr 2024 02:38:26 AM UTC. 2024-04-06T02:40:04.696 INFO:teuthology.orchestra.run.smithi105.stderr:Modular dependency problems: 2024-04-06T02:40:04.696 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-06T02:40:04.696 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 1: conflicting requests 2024-04-06T02:40:04.696 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.696 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 2: conflicting requests 2024-04-06T02:40:04.696 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.697 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 3: conflicting requests 2024-04-06T02:40:04.697 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.697 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 4: conflicting requests 2024-04-06T02:40:04.697 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.697 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 5: conflicting requests 2024-04-06T02:40:04.697 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.697 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 6: conflicting requests 2024-04-06T02:40:04.697 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.697 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 7: conflicting requests 2024-04-06T02:40:04.697 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.697 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 8: conflicting requests 2024-04-06T02:40:04.697 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.697 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 9: conflicting requests 2024-04-06T02:40:04.697 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.698 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 10: conflicting requests 2024-04-06T02:40:04.698 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.698 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 11: conflicting requests 2024-04-06T02:40:04.698 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.698 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 12: conflicting requests 2024-04-06T02:40:04.698 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.698 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 13: conflicting requests 2024-04-06T02:40:04.698 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.698 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 14: conflicting requests 2024-04-06T02:40:04.698 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.698 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 15: conflicting requests 2024-04-06T02:40:04.698 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.698 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 16: conflicting requests 2024-04-06T02:40:04.699 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.699 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 17: conflicting requests 2024-04-06T02:40:04.699 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.699 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 18: conflicting requests 2024-04-06T02:40:04.699 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.699 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 19: conflicting requests 2024-04-06T02:40:04.699 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.699 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 20: conflicting requests 2024-04-06T02:40:04.699 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.699 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 21: conflicting requests 2024-04-06T02:40:04.699 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.699 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 22: conflicting requests 2024-04-06T02:40:04.699 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.699 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 23: conflicting requests 2024-04-06T02:40:04.700 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.700 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 24: conflicting requests 2024-04-06T02:40:04.700 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.700 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 25: conflicting requests 2024-04-06T02:40:04.700 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.700 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 26: conflicting requests 2024-04-06T02:40:04.700 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.700 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 27: conflicting requests 2024-04-06T02:40:04.700 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.700 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 28: conflicting requests 2024-04-06T02:40:04.700 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.700 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 29: conflicting requests 2024-04-06T02:40:04.700 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.700 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 30: conflicting requests 2024-04-06T02:40:04.701 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.701 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 31: conflicting requests 2024-04-06T02:40:04.701 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.701 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 32: conflicting requests 2024-04-06T02:40:04.701 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.701 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 33: conflicting requests 2024-04-06T02:40:04.701 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.701 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 34: conflicting requests 2024-04-06T02:40:04.701 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.701 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 35: conflicting requests 2024-04-06T02:40:04.701 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.701 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 36: conflicting requests 2024-04-06T02:40:04.701 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.701 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 37: conflicting requests 2024-04-06T02:40:04.702 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.702 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.702 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.702 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:40:04.702 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.702 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.702 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.702 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.702 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 39: conflicting requests 2024-04-06T02:40:04.702 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.702 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.702 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.702 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.703 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.703 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.703 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:40:04.703 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.703 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.703 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.703 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.703 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:40:04.703 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.703 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.703 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.703 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.703 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:40:04.703 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.704 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.704 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.704 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.704 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:40:04.704 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.704 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.704 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.704 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.704 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:40:04.704 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.705 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.705 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.705 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.705 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:40:04.705 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.705 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.705 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.705 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:40:04.706 INFO:teuthology.orchestra.run.smithi105.stdout:Package librados2-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-06T02:40:04.766 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-06T02:40:04.767 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-04-06T02:40:04.767 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-06T02:40:04.844 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install librbd1 2024-04-06T02:40:05.226 INFO:teuthology.orchestra.run.smithi184.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 1.0 MB/s | 16 MB 00:15 2024-04-06T02:40:05.226 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:40:05.226 INFO:teuthology.orchestra.run.smithi184.stdout:Total 2.2 MB/s | 38 MB 00:17 2024-04-06T02:40:05.439 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:39 ago on Sat 06 Apr 2024 02:38:26 AM UTC. 2024-04-06T02:40:05.579 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-04-06T02:40:05.606 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-04-06T02:40:05.606 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-04-06T02:40:05.729 INFO:teuthology.orchestra.run.smithi105.stderr:Modular dependency problems: 2024-04-06T02:40:05.729 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-06T02:40:05.729 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 1: conflicting requests 2024-04-06T02:40:05.729 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.729 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 2: conflicting requests 2024-04-06T02:40:05.729 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.729 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 3: conflicting requests 2024-04-06T02:40:05.730 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.730 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 4: conflicting requests 2024-04-06T02:40:05.730 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.730 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 5: conflicting requests 2024-04-06T02:40:05.730 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.730 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 6: conflicting requests 2024-04-06T02:40:05.730 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.730 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 7: conflicting requests 2024-04-06T02:40:05.730 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.730 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 8: conflicting requests 2024-04-06T02:40:05.730 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.730 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 9: conflicting requests 2024-04-06T02:40:05.730 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.731 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 10: conflicting requests 2024-04-06T02:40:05.731 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.731 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 11: conflicting requests 2024-04-06T02:40:05.731 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.731 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 12: conflicting requests 2024-04-06T02:40:05.731 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.731 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 13: conflicting requests 2024-04-06T02:40:05.731 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.731 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 14: conflicting requests 2024-04-06T02:40:05.731 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.731 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 15: conflicting requests 2024-04-06T02:40:05.731 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.731 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 16: conflicting requests 2024-04-06T02:40:05.731 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.732 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 17: conflicting requests 2024-04-06T02:40:05.732 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.732 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 18: conflicting requests 2024-04-06T02:40:05.732 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.732 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 19: conflicting requests 2024-04-06T02:40:05.732 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.732 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 20: conflicting requests 2024-04-06T02:40:05.732 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.732 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 21: conflicting requests 2024-04-06T02:40:05.732 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.732 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 22: conflicting requests 2024-04-06T02:40:05.732 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.732 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 23: conflicting requests 2024-04-06T02:40:05.732 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.733 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 24: conflicting requests 2024-04-06T02:40:05.733 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.733 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 25: conflicting requests 2024-04-06T02:40:05.733 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.733 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 26: conflicting requests 2024-04-06T02:40:05.733 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.733 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 27: conflicting requests 2024-04-06T02:40:05.733 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.733 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 28: conflicting requests 2024-04-06T02:40:05.733 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.733 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 29: conflicting requests 2024-04-06T02:40:05.733 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.733 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 30: conflicting requests 2024-04-06T02:40:05.733 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.734 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 31: conflicting requests 2024-04-06T02:40:05.734 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.734 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 32: conflicting requests 2024-04-06T02:40:05.734 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.734 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 33: conflicting requests 2024-04-06T02:40:05.734 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.734 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 34: conflicting requests 2024-04-06T02:40:05.734 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.734 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 35: conflicting requests 2024-04-06T02:40:05.734 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.734 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 36: conflicting requests 2024-04-06T02:40:05.734 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.734 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 37: conflicting requests 2024-04-06T02:40:05.735 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.735 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.735 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.735 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:40:05.735 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.735 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.735 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.735 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.735 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 39: conflicting requests 2024-04-06T02:40:05.735 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.735 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.735 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.735 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.736 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.736 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.736 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:40:05.736 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.736 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.736 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.736 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.736 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:40:05.736 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.736 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.737 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.737 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.737 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:40:05.737 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.737 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.737 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.737 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.737 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:40:05.737 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.737 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.737 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.737 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.737 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:40:05.737 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.738 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.738 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.738 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.738 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:40:05.738 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.738 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.738 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.738 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:40:05.738 INFO:teuthology.orchestra.run.smithi105.stdout:Package librbd1-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-06T02:40:05.798 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-06T02:40:05.799 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-04-06T02:40:05.799 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-06T02:40:05.877 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-rados 2024-04-06T02:40:06.100 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-04-06T02:40:06.100 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-04-06T02:40:06.467 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:40 ago on Sat 06 Apr 2024 02:38:26 AM UTC. 2024-04-06T02:40:06.759 INFO:teuthology.orchestra.run.smithi105.stderr:Modular dependency problems: 2024-04-06T02:40:06.759 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-06T02:40:06.759 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 1: conflicting requests 2024-04-06T02:40:06.759 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.760 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 2: conflicting requests 2024-04-06T02:40:06.760 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.760 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 3: conflicting requests 2024-04-06T02:40:06.760 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.760 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 4: conflicting requests 2024-04-06T02:40:06.760 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.760 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 5: conflicting requests 2024-04-06T02:40:06.761 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.761 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 6: conflicting requests 2024-04-06T02:40:06.761 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.761 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 7: conflicting requests 2024-04-06T02:40:06.761 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.761 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 8: conflicting requests 2024-04-06T02:40:06.762 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.762 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 9: conflicting requests 2024-04-06T02:40:06.762 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.762 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 10: conflicting requests 2024-04-06T02:40:06.762 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.763 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 11: conflicting requests 2024-04-06T02:40:06.763 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.763 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 12: conflicting requests 2024-04-06T02:40:06.763 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.763 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 13: conflicting requests 2024-04-06T02:40:06.763 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.764 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 14: conflicting requests 2024-04-06T02:40:06.764 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.764 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 15: conflicting requests 2024-04-06T02:40:06.764 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.764 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 16: conflicting requests 2024-04-06T02:40:06.764 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.764 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 17: conflicting requests 2024-04-06T02:40:06.765 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.765 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 18: conflicting requests 2024-04-06T02:40:06.765 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.765 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 19: conflicting requests 2024-04-06T02:40:06.765 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.765 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 20: conflicting requests 2024-04-06T02:40:06.766 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.766 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 21: conflicting requests 2024-04-06T02:40:06.766 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.766 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 22: conflicting requests 2024-04-06T02:40:06.766 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.766 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 23: conflicting requests 2024-04-06T02:40:06.767 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.767 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 24: conflicting requests 2024-04-06T02:40:06.767 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.767 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 25: conflicting requests 2024-04-06T02:40:06.767 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.767 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 26: conflicting requests 2024-04-06T02:40:06.768 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.768 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 27: conflicting requests 2024-04-06T02:40:06.768 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.768 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 28: conflicting requests 2024-04-06T02:40:06.768 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.768 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 29: conflicting requests 2024-04-06T02:40:06.768 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.769 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 30: conflicting requests 2024-04-06T02:40:06.769 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.769 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 31: conflicting requests 2024-04-06T02:40:06.769 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.769 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 32: conflicting requests 2024-04-06T02:40:06.769 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.770 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 33: conflicting requests 2024-04-06T02:40:06.770 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.770 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 34: conflicting requests 2024-04-06T02:40:06.770 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.770 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 35: conflicting requests 2024-04-06T02:40:06.770 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.771 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 36: conflicting requests 2024-04-06T02:40:06.771 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.771 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 37: conflicting requests 2024-04-06T02:40:06.771 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.771 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.771 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.772 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:40:06.772 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.772 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.772 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.772 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.772 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 39: conflicting requests 2024-04-06T02:40:06.773 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.773 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.773 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.773 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.773 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.773 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.774 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:40:06.774 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.774 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.774 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.774 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.775 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:40:06.775 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.775 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.775 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.775 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.775 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:40:06.776 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.776 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.776 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.776 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.776 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:40:06.776 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.777 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.777 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.777 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.777 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:40:06.777 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.777 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.778 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.778 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.778 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:40:06.778 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.778 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.778 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.779 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:40:06.779 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-rados-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-06T02:40:06.827 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-06T02:40:06.828 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-04-06T02:40:06.828 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-06T02:40:06.856 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-04-06T02:40:06.911 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-rgw 2024-04-06T02:40:06.922 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-06T02:40:06.965 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-06T02:40:07.064 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-06T02:40:07.099 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-06T02:40:07.268 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-06T02:40:07.298 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-06T02:40:07.383 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-06T02:40:07.501 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:41 ago on Sat 06 Apr 2024 02:38:26 AM UTC. 2024-04-06T02:40:07.793 INFO:teuthology.orchestra.run.smithi105.stderr:Modular dependency problems: 2024-04-06T02:40:07.794 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-06T02:40:07.794 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 1: conflicting requests 2024-04-06T02:40:07.794 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.794 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 2: conflicting requests 2024-04-06T02:40:07.794 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.795 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 3: conflicting requests 2024-04-06T02:40:07.795 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.795 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 4: conflicting requests 2024-04-06T02:40:07.795 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.795 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 5: conflicting requests 2024-04-06T02:40:07.795 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.795 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 6: conflicting requests 2024-04-06T02:40:07.795 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.796 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 7: conflicting requests 2024-04-06T02:40:07.796 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.796 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 8: conflicting requests 2024-04-06T02:40:07.796 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.796 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 9: conflicting requests 2024-04-06T02:40:07.796 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.796 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 10: conflicting requests 2024-04-06T02:40:07.796 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.797 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 11: conflicting requests 2024-04-06T02:40:07.797 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.797 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 12: conflicting requests 2024-04-06T02:40:07.797 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.797 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 13: conflicting requests 2024-04-06T02:40:07.797 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.797 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 14: conflicting requests 2024-04-06T02:40:07.798 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.798 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 15: conflicting requests 2024-04-06T02:40:07.798 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.798 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 16: conflicting requests 2024-04-06T02:40:07.798 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.798 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 17: conflicting requests 2024-04-06T02:40:07.799 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.799 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 18: conflicting requests 2024-04-06T02:40:07.799 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.799 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 19: conflicting requests 2024-04-06T02:40:07.799 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.799 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 20: conflicting requests 2024-04-06T02:40:07.800 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.800 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 21: conflicting requests 2024-04-06T02:40:07.800 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.800 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 22: conflicting requests 2024-04-06T02:40:07.800 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.800 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 23: conflicting requests 2024-04-06T02:40:07.800 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.800 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 24: conflicting requests 2024-04-06T02:40:07.800 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.801 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 25: conflicting requests 2024-04-06T02:40:07.801 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.801 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 26: conflicting requests 2024-04-06T02:40:07.801 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.801 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 27: conflicting requests 2024-04-06T02:40:07.801 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.801 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 28: conflicting requests 2024-04-06T02:40:07.801 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.801 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 29: conflicting requests 2024-04-06T02:40:07.801 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.801 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 30: conflicting requests 2024-04-06T02:40:07.801 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.801 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 31: conflicting requests 2024-04-06T02:40:07.801 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.802 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 32: conflicting requests 2024-04-06T02:40:07.802 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.802 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 33: conflicting requests 2024-04-06T02:40:07.802 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.802 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 34: conflicting requests 2024-04-06T02:40:07.802 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.802 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 35: conflicting requests 2024-04-06T02:40:07.802 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.802 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 36: conflicting requests 2024-04-06T02:40:07.802 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.802 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 37: conflicting requests 2024-04-06T02:40:07.802 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.803 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.803 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.803 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:40:07.803 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.803 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.803 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.803 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.803 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 39: conflicting requests 2024-04-06T02:40:07.803 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.803 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.803 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.803 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.803 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.803 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.804 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:40:07.804 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.804 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.804 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.804 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.804 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:40:07.804 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.804 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.804 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.804 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.804 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:40:07.804 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.804 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.805 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.805 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.805 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:40:07.805 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.805 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.805 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.805 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.805 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:40:07.805 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.805 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.805 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.805 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.805 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:40:07.805 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.806 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.806 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.806 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:40:07.806 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-rgw-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-06T02:40:07.861 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-06T02:40:07.863 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-04-06T02:40:07.863 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-06T02:40:07.939 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-cephfs 2024-04-06T02:40:08.035 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-06T02:40:08.168 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-06T02:40:08.532 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:42 ago on Sat 06 Apr 2024 02:38:26 AM UTC. 2024-04-06T02:40:08.824 INFO:teuthology.orchestra.run.smithi105.stderr:Modular dependency problems: 2024-04-06T02:40:08.824 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-06T02:40:08.824 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 1: conflicting requests 2024-04-06T02:40:08.824 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.824 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 2: conflicting requests 2024-04-06T02:40:08.824 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.824 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 3: conflicting requests 2024-04-06T02:40:08.824 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.824 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 4: conflicting requests 2024-04-06T02:40:08.824 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.824 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 5: conflicting requests 2024-04-06T02:40:08.824 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.824 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 6: conflicting requests 2024-04-06T02:40:08.825 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.825 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 7: conflicting requests 2024-04-06T02:40:08.825 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.825 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 8: conflicting requests 2024-04-06T02:40:08.825 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.825 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 9: conflicting requests 2024-04-06T02:40:08.825 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.825 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 10: conflicting requests 2024-04-06T02:40:08.825 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.825 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 11: conflicting requests 2024-04-06T02:40:08.825 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.825 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 12: conflicting requests 2024-04-06T02:40:08.825 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.825 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 13: conflicting requests 2024-04-06T02:40:08.825 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.826 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 14: conflicting requests 2024-04-06T02:40:08.826 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.826 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 15: conflicting requests 2024-04-06T02:40:08.826 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.826 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 16: conflicting requests 2024-04-06T02:40:08.826 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.826 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 17: conflicting requests 2024-04-06T02:40:08.826 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.826 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 18: conflicting requests 2024-04-06T02:40:08.826 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.826 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 19: conflicting requests 2024-04-06T02:40:08.826 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.826 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 20: conflicting requests 2024-04-06T02:40:08.826 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.827 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 21: conflicting requests 2024-04-06T02:40:08.827 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.827 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 22: conflicting requests 2024-04-06T02:40:08.827 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.827 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 23: conflicting requests 2024-04-06T02:40:08.827 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.827 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 24: conflicting requests 2024-04-06T02:40:08.827 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.827 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 25: conflicting requests 2024-04-06T02:40:08.827 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.827 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 26: conflicting requests 2024-04-06T02:40:08.827 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.827 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 27: conflicting requests 2024-04-06T02:40:08.827 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.827 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 28: conflicting requests 2024-04-06T02:40:08.828 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.828 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 29: conflicting requests 2024-04-06T02:40:08.828 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.828 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 30: conflicting requests 2024-04-06T02:40:08.828 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.828 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 31: conflicting requests 2024-04-06T02:40:08.828 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.828 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 32: conflicting requests 2024-04-06T02:40:08.828 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.828 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 33: conflicting requests 2024-04-06T02:40:08.828 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.828 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 34: conflicting requests 2024-04-06T02:40:08.828 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.828 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 35: conflicting requests 2024-04-06T02:40:08.828 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.828 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 36: conflicting requests 2024-04-06T02:40:08.829 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.829 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 37: conflicting requests 2024-04-06T02:40:08.829 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.829 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.829 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.829 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:40:08.829 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.829 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.829 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.829 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.829 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 39: conflicting requests 2024-04-06T02:40:08.829 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.829 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.829 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.829 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.830 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.830 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.830 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:40:08.830 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.830 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.830 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.830 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.830 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:40:08.830 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.830 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.830 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.830 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.831 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:40:08.831 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.831 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.831 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.831 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.831 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:40:08.831 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.831 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.831 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.831 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.831 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:40:08.831 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.831 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.831 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.831 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.832 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:40:08.832 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.832 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.832 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.832 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:40:08.832 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-cephfs-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-06T02:40:08.892 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-06T02:40:08.893 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-04-06T02:40:08.893 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-06T02:40:08.968 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-rbd 2024-04-06T02:40:09.394 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-06T02:40:09.561 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:43 ago on Sat 06 Apr 2024 02:38:26 AM UTC. 2024-04-06T02:40:09.846 INFO:teuthology.orchestra.run.smithi105.stderr:Modular dependency problems: 2024-04-06T02:40:09.846 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-06T02:40:09.846 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 1: conflicting requests 2024-04-06T02:40:09.847 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.847 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 2: conflicting requests 2024-04-06T02:40:09.847 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.847 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 3: conflicting requests 2024-04-06T02:40:09.847 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.847 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 4: conflicting requests 2024-04-06T02:40:09.847 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.847 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 5: conflicting requests 2024-04-06T02:40:09.847 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.847 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 6: conflicting requests 2024-04-06T02:40:09.847 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.847 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 7: conflicting requests 2024-04-06T02:40:09.847 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.848 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 8: conflicting requests 2024-04-06T02:40:09.848 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.848 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 9: conflicting requests 2024-04-06T02:40:09.848 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.848 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 10: conflicting requests 2024-04-06T02:40:09.848 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.848 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 11: conflicting requests 2024-04-06T02:40:09.848 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.848 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 12: conflicting requests 2024-04-06T02:40:09.848 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.848 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 13: conflicting requests 2024-04-06T02:40:09.848 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.848 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 14: conflicting requests 2024-04-06T02:40:09.848 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.849 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 15: conflicting requests 2024-04-06T02:40:09.849 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.849 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 16: conflicting requests 2024-04-06T02:40:09.849 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.849 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 17: conflicting requests 2024-04-06T02:40:09.849 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.849 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 18: conflicting requests 2024-04-06T02:40:09.849 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.849 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 19: conflicting requests 2024-04-06T02:40:09.849 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.849 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 20: conflicting requests 2024-04-06T02:40:09.849 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.849 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 21: conflicting requests 2024-04-06T02:40:09.849 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.850 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 22: conflicting requests 2024-04-06T02:40:09.850 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.850 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 23: conflicting requests 2024-04-06T02:40:09.850 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.850 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 24: conflicting requests 2024-04-06T02:40:09.850 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.850 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 25: conflicting requests 2024-04-06T02:40:09.850 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.850 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 26: conflicting requests 2024-04-06T02:40:09.850 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.850 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 27: conflicting requests 2024-04-06T02:40:09.850 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.850 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 28: conflicting requests 2024-04-06T02:40:09.851 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.851 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 29: conflicting requests 2024-04-06T02:40:09.851 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.851 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 30: conflicting requests 2024-04-06T02:40:09.851 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.851 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 31: conflicting requests 2024-04-06T02:40:09.851 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.851 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 32: conflicting requests 2024-04-06T02:40:09.851 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.851 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 33: conflicting requests 2024-04-06T02:40:09.851 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.851 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 34: conflicting requests 2024-04-06T02:40:09.851 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.852 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 35: conflicting requests 2024-04-06T02:40:09.852 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.852 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 36: conflicting requests 2024-04-06T02:40:09.852 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.852 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 37: conflicting requests 2024-04-06T02:40:09.852 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.852 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.852 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.852 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:40:09.852 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.852 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.852 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.852 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.852 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 39: conflicting requests 2024-04-06T02:40:09.852 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.853 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.853 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.853 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.853 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.853 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.853 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:40:09.853 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.853 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.853 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.853 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.853 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:40:09.853 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.853 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.853 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.853 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.854 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:40:09.854 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.854 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.854 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.854 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.854 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:40:09.854 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.854 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.854 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.854 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.854 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:40:09.854 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.854 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.855 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.855 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.855 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:40:09.855 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.855 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.855 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.855 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:40:09.855 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-rbd-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-06T02:40:09.914 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-06T02:40:09.916 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-04-06T02:40:09.916 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-06T02:40:09.989 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install rbd-fuse 2024-04-06T02:40:10.583 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:44 ago on Sat 06 Apr 2024 02:38:26 AM UTC. 2024-04-06T02:40:10.611 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-06T02:40:10.639 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-04-06T02:40:10.876 INFO:teuthology.orchestra.run.smithi105.stderr:Modular dependency problems: 2024-04-06T02:40:10.876 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-06T02:40:10.876 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 1: conflicting requests 2024-04-06T02:40:10.876 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.876 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 2: conflicting requests 2024-04-06T02:40:10.876 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.877 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 3: conflicting requests 2024-04-06T02:40:10.877 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.877 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 4: conflicting requests 2024-04-06T02:40:10.877 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.877 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 5: conflicting requests 2024-04-06T02:40:10.877 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.877 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 6: conflicting requests 2024-04-06T02:40:10.877 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.877 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 7: conflicting requests 2024-04-06T02:40:10.877 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.877 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 8: conflicting requests 2024-04-06T02:40:10.877 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.877 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 9: conflicting requests 2024-04-06T02:40:10.878 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.878 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 10: conflicting requests 2024-04-06T02:40:10.878 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.878 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 11: conflicting requests 2024-04-06T02:40:10.878 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.878 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 12: conflicting requests 2024-04-06T02:40:10.878 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.878 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 13: conflicting requests 2024-04-06T02:40:10.878 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.878 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 14: conflicting requests 2024-04-06T02:40:10.878 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.878 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 15: conflicting requests 2024-04-06T02:40:10.879 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.879 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 16: conflicting requests 2024-04-06T02:40:10.879 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.879 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 17: conflicting requests 2024-04-06T02:40:10.879 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.879 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 18: conflicting requests 2024-04-06T02:40:10.879 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.879 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 19: conflicting requests 2024-04-06T02:40:10.879 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.879 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 20: conflicting requests 2024-04-06T02:40:10.879 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.879 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 21: conflicting requests 2024-04-06T02:40:10.879 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.880 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 22: conflicting requests 2024-04-06T02:40:10.880 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.880 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 23: conflicting requests 2024-04-06T02:40:10.880 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.880 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 24: conflicting requests 2024-04-06T02:40:10.880 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.880 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 25: conflicting requests 2024-04-06T02:40:10.880 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.880 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 26: conflicting requests 2024-04-06T02:40:10.880 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.880 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 27: conflicting requests 2024-04-06T02:40:10.880 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.880 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 28: conflicting requests 2024-04-06T02:40:10.881 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.881 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 29: conflicting requests 2024-04-06T02:40:10.881 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.881 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 30: conflicting requests 2024-04-06T02:40:10.881 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.881 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 31: conflicting requests 2024-04-06T02:40:10.881 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.881 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 32: conflicting requests 2024-04-06T02:40:10.881 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.881 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 33: conflicting requests 2024-04-06T02:40:10.881 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.881 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 34: conflicting requests 2024-04-06T02:40:10.881 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.881 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 35: conflicting requests 2024-04-06T02:40:10.882 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.882 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 36: conflicting requests 2024-04-06T02:40:10.882 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.882 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 37: conflicting requests 2024-04-06T02:40:10.882 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.882 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.882 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.882 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:40:10.882 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.882 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.882 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.882 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.883 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 39: conflicting requests 2024-04-06T02:40:10.883 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.883 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.883 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.883 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.883 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.883 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.883 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:40:10.883 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.883 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.883 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.884 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.884 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:40:10.884 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.884 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.884 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.884 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.884 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:40:10.884 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.884 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.884 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.884 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.885 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:40:10.885 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.885 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.885 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.885 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.885 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:40:10.885 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.885 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.885 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.885 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.885 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:40:10.885 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.885 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.886 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.886 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:40:10.944 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-06T02:40:10.945 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:40:10.945 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2024-04-06T02:40:10.945 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:40:10.945 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-06T02:40:10.945 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse x86_64 2:18.2.1-0.el9 ceph 86 k 2024-04-06T02:40:10.945 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:40:10.945 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-06T02:40:10.945 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:40:10.945 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-04-06T02:40:10.946 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:40:10.946 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 86 k 2024-04-06T02:40:10.946 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 226 k 2024-04-06T02:40:10.946 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-06T02:40:11.041 INFO:teuthology.orchestra.run.smithi105.stdout:rbd-fuse-18.2.1-0.el9.x86_64.rpm 905 kB/s | 86 kB 00:00 2024-04-06T02:40:11.042 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:40:11.042 INFO:teuthology.orchestra.run.smithi105.stdout:Total 892 kB/s | 86 kB 00:00 2024-04-06T02:40:11.042 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-06T02:40:11.049 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-06T02:40:11.049 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-06T02:40:11.107 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-06T02:40:11.107 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-06T02:40:11.245 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-06T02:40:11.348 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:11.773 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:12.079 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:12.079 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:40:12.079 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-06T02:40:12.079 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse-2:18.2.1-0.el9.x86_64 2024-04-06T02:40:12.079 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:40:12.079 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-06T02:40:12.307 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install rbd-mirror 2024-04-06T02:40:12.919 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:46 ago on Sat 06 Apr 2024 02:38:26 AM UTC. 2024-04-06T02:40:13.212 INFO:teuthology.orchestra.run.smithi105.stderr:Modular dependency problems: 2024-04-06T02:40:13.212 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-06T02:40:13.212 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 1: conflicting requests 2024-04-06T02:40:13.212 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.213 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 2: conflicting requests 2024-04-06T02:40:13.213 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.213 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 3: conflicting requests 2024-04-06T02:40:13.213 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.213 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 4: conflicting requests 2024-04-06T02:40:13.213 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.213 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 5: conflicting requests 2024-04-06T02:40:13.213 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.213 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 6: conflicting requests 2024-04-06T02:40:13.213 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.213 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 7: conflicting requests 2024-04-06T02:40:13.213 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.213 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 8: conflicting requests 2024-04-06T02:40:13.213 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.213 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 9: conflicting requests 2024-04-06T02:40:13.214 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.214 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 10: conflicting requests 2024-04-06T02:40:13.214 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.214 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 11: conflicting requests 2024-04-06T02:40:13.214 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.214 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 12: conflicting requests 2024-04-06T02:40:13.214 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.214 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 13: conflicting requests 2024-04-06T02:40:13.214 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.214 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 14: conflicting requests 2024-04-06T02:40:13.214 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.214 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 15: conflicting requests 2024-04-06T02:40:13.214 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.214 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 16: conflicting requests 2024-04-06T02:40:13.215 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.215 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 17: conflicting requests 2024-04-06T02:40:13.215 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.215 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 18: conflicting requests 2024-04-06T02:40:13.215 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.215 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 19: conflicting requests 2024-04-06T02:40:13.215 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.215 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 20: conflicting requests 2024-04-06T02:40:13.215 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.215 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 21: conflicting requests 2024-04-06T02:40:13.215 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.215 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 22: conflicting requests 2024-04-06T02:40:13.215 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.215 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 23: conflicting requests 2024-04-06T02:40:13.216 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.216 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 24: conflicting requests 2024-04-06T02:40:13.216 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.216 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 25: conflicting requests 2024-04-06T02:40:13.216 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.216 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 26: conflicting requests 2024-04-06T02:40:13.216 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.216 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 27: conflicting requests 2024-04-06T02:40:13.216 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.216 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 28: conflicting requests 2024-04-06T02:40:13.216 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.216 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 29: conflicting requests 2024-04-06T02:40:13.216 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.217 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 30: conflicting requests 2024-04-06T02:40:13.217 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.217 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 31: conflicting requests 2024-04-06T02:40:13.217 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.217 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 32: conflicting requests 2024-04-06T02:40:13.217 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.217 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 33: conflicting requests 2024-04-06T02:40:13.217 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.217 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 34: conflicting requests 2024-04-06T02:40:13.217 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.217 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 35: conflicting requests 2024-04-06T02:40:13.217 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.218 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 36: conflicting requests 2024-04-06T02:40:13.218 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.218 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 37: conflicting requests 2024-04-06T02:40:13.218 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.218 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.218 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.218 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:40:13.218 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.218 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.218 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.218 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.218 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 39: conflicting requests 2024-04-06T02:40:13.218 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.219 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.219 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.219 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.219 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.219 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.219 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:40:13.219 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.219 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.219 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.219 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.219 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:40:13.219 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.220 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.220 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.220 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.220 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:40:13.220 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.220 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.220 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.220 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.220 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:40:13.220 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.220 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.220 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.220 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.220 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:40:13.220 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.221 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.221 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.221 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.221 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:40:13.221 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.221 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.221 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.221 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:40:13.227 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-04-06T02:40:13.227 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 1/12 2024-04-06T02:40:13.227 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-06T02:40:13.227 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-06T02:40:13.227 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-06T02:40:13.227 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-06T02:40:13.227 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-06T02:40:13.228 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-06T02:40:13.228 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-06T02:40:13.228 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 9/12 2024-04-06T02:40:13.228 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-06T02:40:13.228 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-06T02:40:13.280 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-06T02:40:13.281 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:40:13.281 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2024-04-06T02:40:13.281 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:40:13.282 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-06T02:40:13.282 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-04-06T02:40:13.282 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:40:13.282 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-06T02:40:13.282 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:40:13.282 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-04-06T02:40:13.282 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:40:13.282 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 3.0 M 2024-04-06T02:40:13.282 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 12 M 2024-04-06T02:40:13.282 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-06T02:40:13.444 INFO:teuthology.orchestra.run.smithi105.stdout:rbd-mirror-18.2.1-0.el9.x86_64.rpm 18 MB/s | 3.0 MB 00:00 2024-04-06T02:40:13.445 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:40:13.445 INFO:teuthology.orchestra.run.smithi105.stdout:Total 18 MB/s | 3.0 MB 00:00 2024-04-06T02:40:13.446 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-06T02:40:13.454 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-06T02:40:13.454 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-06T02:40:13.531 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-06T02:40:13.531 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-06T02:40:13.672 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-06T02:40:13.672 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:13.673 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-04-06T02:40:13.673 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarch 2024-04-06T02:40:13.673 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-06T02:40:13.673 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-06T02:40:13.673 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-06T02:40:13.673 INFO:teuthology.orchestra.run.smithi184.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-06T02:40:13.673 INFO:teuthology.orchestra.run.smithi184.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-06T02:40:13.673 INFO:teuthology.orchestra.run.smithi184.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-06T02:40:13.673 INFO:teuthology.orchestra.run.smithi184.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-06T02:40:13.673 INFO:teuthology.orchestra.run.smithi184.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-06T02:40:13.673 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-06T02:40:13.673 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-06T02:40:13.673 INFO:teuthology.orchestra.run.smithi184.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-06T02:40:13.673 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:13.674 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-04-06T02:40:13.946 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-06T02:40:14.020 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:14.044 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:14.044 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-06T02:40:14.044 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-06T02:40:14.044 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-06T02:40:14.045 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-06T02:40:14.045 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:40:14.144 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-rook 2024-04-06T02:40:14.750 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:36 ago on Sat 06 Apr 2024 02:38:38 AM UTC. 2024-04-06T02:40:15.038 INFO:teuthology.orchestra.run.smithi184.stderr:Modular dependency problems: 2024-04-06T02:40:15.038 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-04-06T02:40:15.038 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 1: conflicting requests 2024-04-06T02:40:15.038 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.038 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 2: conflicting requests 2024-04-06T02:40:15.038 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.038 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 3: conflicting requests 2024-04-06T02:40:15.038 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.039 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 4: conflicting requests 2024-04-06T02:40:15.039 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.039 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 5: conflicting requests 2024-04-06T02:40:15.039 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.039 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 6: conflicting requests 2024-04-06T02:40:15.039 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.039 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 7: conflicting requests 2024-04-06T02:40:15.039 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.039 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 8: conflicting requests 2024-04-06T02:40:15.039 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.039 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 9: conflicting requests 2024-04-06T02:40:15.039 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.039 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 10: conflicting requests 2024-04-06T02:40:15.039 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.040 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 11: conflicting requests 2024-04-06T02:40:15.040 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.040 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 12: conflicting requests 2024-04-06T02:40:15.040 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.040 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 13: conflicting requests 2024-04-06T02:40:15.040 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.040 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 14: conflicting requests 2024-04-06T02:40:15.040 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.040 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 15: conflicting requests 2024-04-06T02:40:15.040 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.040 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 16: conflicting requests 2024-04-06T02:40:15.040 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.041 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 17: conflicting requests 2024-04-06T02:40:15.041 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.041 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 18: conflicting requests 2024-04-06T02:40:15.041 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.041 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 19: conflicting requests 2024-04-06T02:40:15.041 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.041 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 20: conflicting requests 2024-04-06T02:40:15.041 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.041 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 21: conflicting requests 2024-04-06T02:40:15.041 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.041 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 22: conflicting requests 2024-04-06T02:40:15.041 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.041 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 23: conflicting requests 2024-04-06T02:40:15.042 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.042 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 24: conflicting requests 2024-04-06T02:40:15.042 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.042 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 25: conflicting requests 2024-04-06T02:40:15.042 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.042 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 26: conflicting requests 2024-04-06T02:40:15.042 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.042 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 27: conflicting requests 2024-04-06T02:40:15.042 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.042 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 28: conflicting requests 2024-04-06T02:40:15.042 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.042 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 29: conflicting requests 2024-04-06T02:40:15.043 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.043 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 30: conflicting requests 2024-04-06T02:40:15.043 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.043 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 31: conflicting requests 2024-04-06T02:40:15.043 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.043 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 32: conflicting requests 2024-04-06T02:40:15.043 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.043 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 33: conflicting requests 2024-04-06T02:40:15.043 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.043 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 34: conflicting requests 2024-04-06T02:40:15.043 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.043 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 35: conflicting requests 2024-04-06T02:40:15.043 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.043 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 36: conflicting requests 2024-04-06T02:40:15.044 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.044 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 37: conflicting requests 2024-04-06T02:40:15.044 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.044 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.044 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.044 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:40:15.044 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.044 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.044 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.044 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.044 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 39: conflicting requests 2024-04-06T02:40:15.044 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.044 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.044 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.045 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.045 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.045 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.045 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:40:15.045 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.045 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.045 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.045 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.045 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:40:15.045 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.045 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.045 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.045 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.045 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:40:15.046 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.046 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.046 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.046 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.046 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:40:15.046 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.046 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.046 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.046 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.046 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:40:15.046 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.046 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.046 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.047 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.047 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:40:15.047 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.047 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.047 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.047 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:40:15.108 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-04-06T02:40:15.109 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:40:15.109 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-04-06T02:40:15.109 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:40:15.110 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-04-06T02:40:15.110 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-rook noarch 2:18.2.1-0.el9 ceph-noarch 50 k 2024-04-06T02:40:15.110 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-04-06T02:40:15.110 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-06T02:40:15.110 INFO:teuthology.orchestra.run.smithi184.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-06T02:40:15.110 INFO:teuthology.orchestra.run.smithi184.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-06T02:40:15.110 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-06T02:40:15.110 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-06T02:40:15.110 INFO:teuthology.orchestra.run.smithi184.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-06T02:40:15.110 INFO:teuthology.orchestra.run.smithi184.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-06T02:40:15.111 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-06T02:40:15.111 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-06T02:40:15.111 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-06T02:40:15.111 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-06T02:40:15.111 INFO:teuthology.orchestra.run.smithi184.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-06T02:40:15.111 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:15.111 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-04-06T02:40:15.111 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:40:15.111 INFO:teuthology.orchestra.run.smithi184.stdout:Install 13 Packages 2024-04-06T02:40:15.111 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:15.112 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 2.2 M 2024-04-06T02:40:15.112 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 26 M 2024-04-06T02:40:15.112 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-04-06T02:40:15.225 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:15.225 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:40:15.225 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-06T02:40:15.225 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror-2:18.2.1-0.el9.x86_64 2024-04-06T02:40:15.225 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:40:15.226 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-06T02:40:15.520 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install rbd-nbd 2024-04-06T02:40:15.529 INFO:teuthology.orchestra.run.smithi184.stdout:(1/13): ceph-mgr-rook-18.2.1-0.el9.noarch.rpm 814 kB/s | 50 kB 00:00 2024-04-06T02:40:15.546 INFO:teuthology.orchestra.run.smithi184.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 244 kB/s | 19 kB 00:00 2024-04-06T02:40:15.563 INFO:teuthology.orchestra.run.smithi184.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 269 kB/s | 26 kB 00:00 2024-04-06T02:40:15.647 INFO:teuthology.orchestra.run.smithi184.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 159 kB 00:00 2024-04-06T02:40:15.672 INFO:teuthology.orchestra.run.smithi184.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.5 MB/s | 279 kB 00:00 2024-04-06T02:40:15.689 INFO:teuthology.orchestra.run.smithi184.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.3 MB/s | 54 kB 00:00 2024-04-06T02:40:15.714 INFO:teuthology.orchestra.run.smithi184.stdout:(7/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.2 MB/s | 222 kB 00:00 2024-04-06T02:40:15.831 INFO:teuthology.orchestra.run.smithi184.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 203 kB/s | 32 kB 00:00 2024-04-06T02:40:15.856 INFO:teuthology.orchestra.run.smithi184.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 90 kB/s | 15 kB 00:00 2024-04-06T02:40:15.957 INFO:teuthology.orchestra.run.smithi184.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.0 MB/s | 203 kB 00:00 2024-04-06T02:40:15.974 INFO:teuthology.orchestra.run.smithi184.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 506 kB/s | 59 kB 00:00 2024-04-06T02:40:16.032 INFO:teuthology.orchestra.run.smithi184.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 5.1 MB/s | 1.0 MB 00:00 2024-04-06T02:40:16.057 INFO:teuthology.orchestra.run.smithi184.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 1.1 MB/s | 90 kB 00:00 2024-04-06T02:40:16.059 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:40:16.060 INFO:teuthology.orchestra.run.smithi184.stdout:Total 2.3 MB/s | 2.2 MB 00:00 2024-04-06T02:40:16.121 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-04-06T02:40:16.128 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:50 ago on Sat 06 Apr 2024 02:38:26 AM UTC. 2024-04-06T02:40:16.136 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-04-06T02:40:16.136 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-04-06T02:40:16.229 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-04-06T02:40:16.231 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-04-06T02:40:16.415 INFO:teuthology.orchestra.run.smithi105.stderr:Modular dependency problems: 2024-04-06T02:40:16.416 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-06T02:40:16.416 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 1: conflicting requests 2024-04-06T02:40:16.416 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.416 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 2: conflicting requests 2024-04-06T02:40:16.416 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.416 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 3: conflicting requests 2024-04-06T02:40:16.416 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.416 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 4: conflicting requests 2024-04-06T02:40:16.416 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.416 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 5: conflicting requests 2024-04-06T02:40:16.416 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.416 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 6: conflicting requests 2024-04-06T02:40:16.417 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.417 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 7: conflicting requests 2024-04-06T02:40:16.417 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.417 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 8: conflicting requests 2024-04-06T02:40:16.417 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.417 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 9: conflicting requests 2024-04-06T02:40:16.417 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.417 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 10: conflicting requests 2024-04-06T02:40:16.417 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.417 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 11: conflicting requests 2024-04-06T02:40:16.417 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.417 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 12: conflicting requests 2024-04-06T02:40:16.417 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.417 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 13: conflicting requests 2024-04-06T02:40:16.418 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.418 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 14: conflicting requests 2024-04-06T02:40:16.418 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.418 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 15: conflicting requests 2024-04-06T02:40:16.418 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.418 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 16: conflicting requests 2024-04-06T02:40:16.418 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.418 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 17: conflicting requests 2024-04-06T02:40:16.418 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.418 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 18: conflicting requests 2024-04-06T02:40:16.418 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.418 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 19: conflicting requests 2024-04-06T02:40:16.418 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.419 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 20: conflicting requests 2024-04-06T02:40:16.419 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.419 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 21: conflicting requests 2024-04-06T02:40:16.419 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.419 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 22: conflicting requests 2024-04-06T02:40:16.419 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.419 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 23: conflicting requests 2024-04-06T02:40:16.419 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.419 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 24: conflicting requests 2024-04-06T02:40:16.419 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.419 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 25: conflicting requests 2024-04-06T02:40:16.419 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.419 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 26: conflicting requests 2024-04-06T02:40:16.420 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.420 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 27: conflicting requests 2024-04-06T02:40:16.420 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.420 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 28: conflicting requests 2024-04-06T02:40:16.420 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.420 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 29: conflicting requests 2024-04-06T02:40:16.420 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.420 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 30: conflicting requests 2024-04-06T02:40:16.420 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.420 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 31: conflicting requests 2024-04-06T02:40:16.420 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.420 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 32: conflicting requests 2024-04-06T02:40:16.420 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.421 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 33: conflicting requests 2024-04-06T02:40:16.421 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.421 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 34: conflicting requests 2024-04-06T02:40:16.421 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.421 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 35: conflicting requests 2024-04-06T02:40:16.421 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.421 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 36: conflicting requests 2024-04-06T02:40:16.421 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.421 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 37: conflicting requests 2024-04-06T02:40:16.421 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.421 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.421 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.421 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:40:16.421 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.422 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.422 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.422 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.422 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 39: conflicting requests 2024-04-06T02:40:16.422 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.422 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.422 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.422 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.422 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.422 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.422 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:40:16.422 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.422 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.423 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.423 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.423 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:40:16.423 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.423 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.423 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.423 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.423 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:40:16.423 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.423 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.423 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.423 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.423 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:40:16.424 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.424 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.424 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.424 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.424 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:40:16.424 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.424 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.424 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.424 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.424 INFO:teuthology.orchestra.run.smithi105.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:40:16.424 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.424 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.424 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.424 INFO:teuthology.orchestra.run.smithi105.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:40:16.485 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-06T02:40:16.485 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:40:16.485 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2024-04-06T02:40:16.486 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:40:16.486 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-06T02:40:16.486 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd x86_64 2:18.2.1-0.el9 ceph 171 k 2024-04-06T02:40:16.486 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:40:16.486 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-06T02:40:16.486 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-06T02:40:16.486 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-04-06T02:40:16.486 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:40:16.486 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 171 k 2024-04-06T02:40:16.486 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 494 k 2024-04-06T02:40:16.486 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-06T02:40:16.565 INFO:teuthology.orchestra.run.smithi105.stdout:rbd-nbd-18.2.1-0.el9.x86_64.rpm 2.1 MB/s | 171 kB 00:00 2024-04-06T02:40:16.566 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:40:16.566 INFO:teuthology.orchestra.run.smithi105.stdout:Total 2.1 MB/s | 171 kB 00:00 2024-04-06T02:40:16.566 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-06T02:40:16.574 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-06T02:40:16.574 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-06T02:40:16.632 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-06T02:40:16.633 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-06T02:40:16.740 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-04-06T02:40:16.777 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-06T02:40:16.880 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:16.904 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-06T02:40:16.958 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-06T02:40:17.022 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-06T02:40:17.060 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-06T02:40:17.101 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-06T02:40:17.195 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-06T02:40:17.296 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-06T02:40:17.310 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:17.344 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-06T02:40:17.612 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:17.612 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:40:17.612 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-06T02:40:17.612 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd-2:18.2.1-0.el9.x86_64 2024-04-06T02:40:17.612 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:40:17.612 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-06T02:40:17.859 DEBUG:teuthology.parallel:result is None 2024-04-06T02:40:18.063 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-06T02:40:18.108 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-06T02:40:18.141 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-06T02:40:18.188 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-06T02:40:18.211 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-04-06T02:40:18.980 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-04-06T02:40:18.980 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-rook-2:18.2.1-0.el9.noarch 1/13 2024-04-06T02:40:18.980 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-06T02:40:18.980 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-06T02:40:18.980 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-06T02:40:18.980 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-06T02:40:18.980 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-06T02:40:18.980 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-06T02:40:18.980 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-06T02:40:18.980 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-06T02:40:18.980 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-06T02:40:18.980 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-06T02:40:18.980 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-06T02:40:19.353 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-06T02:40:19.353 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:19.354 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-04-06T02:40:19.354 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-rook-2:18.2.1-0.el9.noarch 2024-04-06T02:40:19.354 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-06T02:40:19.354 INFO:teuthology.orchestra.run.smithi184.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-06T02:40:19.354 INFO:teuthology.orchestra.run.smithi184.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-06T02:40:19.354 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-06T02:40:19.354 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-06T02:40:19.354 INFO:teuthology.orchestra.run.smithi184.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-06T02:40:19.354 INFO:teuthology.orchestra.run.smithi184.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-06T02:40:19.354 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-06T02:40:19.354 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-06T02:40:19.354 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-06T02:40:19.355 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-06T02:40:19.355 INFO:teuthology.orchestra.run.smithi184.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-06T02:40:19.355 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:19.355 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-04-06T02:40:19.630 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-cephadm 2024-04-06T02:40:20.225 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:42 ago on Sat 06 Apr 2024 02:38:38 AM UTC. 2024-04-06T02:40:20.514 INFO:teuthology.orchestra.run.smithi184.stderr:Modular dependency problems: 2024-04-06T02:40:20.514 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-04-06T02:40:20.514 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 1: conflicting requests 2024-04-06T02:40:20.514 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.514 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 2: conflicting requests 2024-04-06T02:40:20.514 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.514 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 3: conflicting requests 2024-04-06T02:40:20.514 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.514 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 4: conflicting requests 2024-04-06T02:40:20.515 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.515 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 5: conflicting requests 2024-04-06T02:40:20.515 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.515 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 6: conflicting requests 2024-04-06T02:40:20.515 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.515 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 7: conflicting requests 2024-04-06T02:40:20.515 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.515 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 8: conflicting requests 2024-04-06T02:40:20.515 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.515 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 9: conflicting requests 2024-04-06T02:40:20.515 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.515 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 10: conflicting requests 2024-04-06T02:40:20.515 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.516 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 11: conflicting requests 2024-04-06T02:40:20.516 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.516 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 12: conflicting requests 2024-04-06T02:40:20.516 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.516 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 13: conflicting requests 2024-04-06T02:40:20.516 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.516 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 14: conflicting requests 2024-04-06T02:40:20.516 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.516 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 15: conflicting requests 2024-04-06T02:40:20.516 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.516 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 16: conflicting requests 2024-04-06T02:40:20.516 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.516 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 17: conflicting requests 2024-04-06T02:40:20.517 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.517 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 18: conflicting requests 2024-04-06T02:40:20.517 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.517 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 19: conflicting requests 2024-04-06T02:40:20.517 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.517 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 20: conflicting requests 2024-04-06T02:40:20.517 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.517 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 21: conflicting requests 2024-04-06T02:40:20.517 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.517 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 22: conflicting requests 2024-04-06T02:40:20.517 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.517 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 23: conflicting requests 2024-04-06T02:40:20.517 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.518 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 24: conflicting requests 2024-04-06T02:40:20.518 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.518 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 25: conflicting requests 2024-04-06T02:40:20.518 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.518 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 26: conflicting requests 2024-04-06T02:40:20.518 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.518 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 27: conflicting requests 2024-04-06T02:40:20.518 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.518 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 28: conflicting requests 2024-04-06T02:40:20.518 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.518 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 29: conflicting requests 2024-04-06T02:40:20.518 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.518 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 30: conflicting requests 2024-04-06T02:40:20.519 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.519 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 31: conflicting requests 2024-04-06T02:40:20.519 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.519 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 32: conflicting requests 2024-04-06T02:40:20.519 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.519 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 33: conflicting requests 2024-04-06T02:40:20.519 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.519 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 34: conflicting requests 2024-04-06T02:40:20.519 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.519 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 35: conflicting requests 2024-04-06T02:40:20.519 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.519 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 36: conflicting requests 2024-04-06T02:40:20.519 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.519 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 37: conflicting requests 2024-04-06T02:40:20.520 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.520 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.520 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.520 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:40:20.520 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.520 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.520 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.520 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.520 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 39: conflicting requests 2024-04-06T02:40:20.520 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.520 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.520 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.520 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.520 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.520 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.521 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:40:20.521 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.521 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.521 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.521 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.521 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:40:20.521 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.521 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.521 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.521 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.521 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:40:20.521 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.521 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.521 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.522 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.522 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:40:20.522 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.522 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.522 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.522 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.522 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:40:20.522 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.522 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.522 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.522 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.522 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:40:20.523 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.523 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.523 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.523 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:40:20.582 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-04-06T02:40:20.583 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:40:20.583 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-04-06T02:40:20.583 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:40:20.583 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-04-06T02:40:20.584 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-cephadm noarch 2:18.2.1-0.el9 ceph-noarch 132 k 2024-04-06T02:40:20.584 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-04-06T02:40:20.584 INFO:teuthology.orchestra.run.smithi184.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-06T02:40:20.584 INFO:teuthology.orchestra.run.smithi184.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-06T02:40:20.584 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-06T02:40:20.584 INFO:teuthology.orchestra.run.smithi184.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-06T02:40:20.584 INFO:teuthology.orchestra.run.smithi184.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-06T02:40:20.584 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:20.584 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-04-06T02:40:20.584 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:40:20.584 INFO:teuthology.orchestra.run.smithi184.stdout:Install 6 Packages 2024-04-06T02:40:20.584 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:20.585 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 7.0 M 2024-04-06T02:40:20.585 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 33 M 2024-04-06T02:40:20.585 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-04-06T02:40:21.159 INFO:teuthology.orchestra.run.smithi184.stdout:(1/6): ceph-mgr-cephadm-18.2.1-0.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-04-06T02:40:21.217 INFO:teuthology.orchestra.run.smithi184.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.9 MB/s | 248 kB 00:00 2024-04-06T02:40:21.318 INFO:teuthology.orchestra.run.smithi184.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 579 kB/s | 58 kB 00:00 2024-04-06T02:40:21.351 INFO:teuthology.orchestra.run.smithi184.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.8 MB/s | 548 kB 00:00 2024-04-06T02:40:21.376 INFO:teuthology.orchestra.run.smithi184.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 870 kB/s | 51 kB 00:00 2024-04-06T02:40:21.460 INFO:teuthology.orchestra.run.smithi184.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-04-06T02:40:21.461 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:40:21.462 INFO:teuthology.orchestra.run.smithi184.stdout:Total 8.0 MB/s | 7.0 MB 00:00 2024-04-06T02:40:21.530 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-04-06T02:40:21.541 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-04-06T02:40:21.542 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-04-06T02:40:21.657 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-04-06T02:40:21.658 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-04-06T02:40:21.995 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-04-06T02:40:22.164 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-06T02:40:22.220 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-06T02:40:22.729 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-06T02:40:22.824 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-06T02:40:22.882 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-06T02:40:22.907 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-04-06T02:40:23.672 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-04-06T02:40:23.672 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 1/6 2024-04-06T02:40:23.672 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-06T02:40:23.673 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-06T02:40:23.673 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-06T02:40:23.673 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-06T02:40:23.994 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-06T02:40:23.995 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:23.995 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-04-06T02:40:23.995 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 2024-04-06T02:40:23.995 INFO:teuthology.orchestra.run.smithi184.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-06T02:40:23.995 INFO:teuthology.orchestra.run.smithi184.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-06T02:40:23.995 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-06T02:40:23.995 INFO:teuthology.orchestra.run.smithi184.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-06T02:40:23.995 INFO:teuthology.orchestra.run.smithi184.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-06T02:40:23.995 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:23.995 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-04-06T02:40:24.275 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-fuse 2024-04-06T02:40:24.864 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:46 ago on Sat 06 Apr 2024 02:38:38 AM UTC. 2024-04-06T02:40:25.150 INFO:teuthology.orchestra.run.smithi184.stderr:Modular dependency problems: 2024-04-06T02:40:25.150 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-04-06T02:40:25.150 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 1: conflicting requests 2024-04-06T02:40:25.151 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.151 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 2: conflicting requests 2024-04-06T02:40:25.151 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.151 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 3: conflicting requests 2024-04-06T02:40:25.151 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.151 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 4: conflicting requests 2024-04-06T02:40:25.151 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.151 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 5: conflicting requests 2024-04-06T02:40:25.151 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.151 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 6: conflicting requests 2024-04-06T02:40:25.151 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.151 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 7: conflicting requests 2024-04-06T02:40:25.151 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.152 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 8: conflicting requests 2024-04-06T02:40:25.152 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.152 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 9: conflicting requests 2024-04-06T02:40:25.152 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.152 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 10: conflicting requests 2024-04-06T02:40:25.152 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.152 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 11: conflicting requests 2024-04-06T02:40:25.152 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.152 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 12: conflicting requests 2024-04-06T02:40:25.152 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.152 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 13: conflicting requests 2024-04-06T02:40:25.152 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.152 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 14: conflicting requests 2024-04-06T02:40:25.153 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.153 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 15: conflicting requests 2024-04-06T02:40:25.153 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.153 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 16: conflicting requests 2024-04-06T02:40:25.153 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.153 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 17: conflicting requests 2024-04-06T02:40:25.153 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.153 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 18: conflicting requests 2024-04-06T02:40:25.153 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.153 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 19: conflicting requests 2024-04-06T02:40:25.153 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.153 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 20: conflicting requests 2024-04-06T02:40:25.153 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.154 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 21: conflicting requests 2024-04-06T02:40:25.154 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.154 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 22: conflicting requests 2024-04-06T02:40:25.154 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.154 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 23: conflicting requests 2024-04-06T02:40:25.154 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.154 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 24: conflicting requests 2024-04-06T02:40:25.154 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.154 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 25: conflicting requests 2024-04-06T02:40:25.154 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.154 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 26: conflicting requests 2024-04-06T02:40:25.154 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.154 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 27: conflicting requests 2024-04-06T02:40:25.154 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.155 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 28: conflicting requests 2024-04-06T02:40:25.155 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.155 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 29: conflicting requests 2024-04-06T02:40:25.155 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.155 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 30: conflicting requests 2024-04-06T02:40:25.155 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.155 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 31: conflicting requests 2024-04-06T02:40:25.155 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.155 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 32: conflicting requests 2024-04-06T02:40:25.155 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.155 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 33: conflicting requests 2024-04-06T02:40:25.155 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.155 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 34: conflicting requests 2024-04-06T02:40:25.156 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.156 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 35: conflicting requests 2024-04-06T02:40:25.156 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.156 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 36: conflicting requests 2024-04-06T02:40:25.156 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.156 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 37: conflicting requests 2024-04-06T02:40:25.156 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.156 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.156 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.156 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:40:25.156 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.156 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.156 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.156 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.157 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 39: conflicting requests 2024-04-06T02:40:25.157 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.157 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.157 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.157 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.157 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.157 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.157 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:40:25.157 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.158 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.158 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.158 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.158 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:40:25.158 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.158 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.158 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.158 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.158 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:40:25.158 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.158 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.158 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.158 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.159 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:40:25.159 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.159 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.159 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.159 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.159 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:40:25.159 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.159 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.159 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.159 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.159 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:40:25.159 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.159 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.159 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.160 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:40:25.220 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-04-06T02:40:25.220 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:40:25.220 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-04-06T02:40:25.220 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:40:25.220 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-04-06T02:40:25.221 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-fuse x86_64 2:18.2.1-0.el9 ceph 839 k 2024-04-06T02:40:25.221 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-04-06T02:40:25.221 INFO:teuthology.orchestra.run.smithi184.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-06T02:40:25.221 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:25.221 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-04-06T02:40:25.221 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:40:25.221 INFO:teuthology.orchestra.run.smithi184.stdout:Install 2 Packages 2024-04-06T02:40:25.221 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:25.222 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 920 k 2024-04-06T02:40:25.222 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 2.7 M 2024-04-06T02:40:25.222 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-04-06T02:40:25.561 INFO:teuthology.orchestra.run.smithi184.stdout:(1/2): ceph-fuse-18.2.1-0.el9.x86_64.rpm 12 MB/s | 839 kB 00:00 2024-04-06T02:40:25.586 INFO:teuthology.orchestra.run.smithi184.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 841 kB/s | 80 kB 00:00 2024-04-06T02:40:25.587 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:40:25.587 INFO:teuthology.orchestra.run.smithi184.stdout:Total 2.5 MB/s | 920 kB 00:00 2024-04-06T02:40:25.598 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-04-06T02:40:25.620 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-04-06T02:40:25.620 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-04-06T02:40:25.696 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-04-06T02:40:25.696 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-04-06T02:40:25.891 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-04-06T02:40:26.016 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-06T02:40:26.085 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-04-06T02:40:26.752 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-04-06T02:40:26.752 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-fuse-2:18.2.1-0.el9.x86_64 1/2 2024-04-06T02:40:27.083 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-06T02:40:27.083 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:27.083 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-04-06T02:40:27.083 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-fuse-2:18.2.1-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-06T02:40:27.083 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:27.084 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-04-06T02:40:27.356 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install librados-devel 2024-04-06T02:40:27.967 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:49 ago on Sat 06 Apr 2024 02:38:38 AM UTC. 2024-04-06T02:40:28.256 INFO:teuthology.orchestra.run.smithi184.stderr:Modular dependency problems: 2024-04-06T02:40:28.257 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-04-06T02:40:28.257 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 1: conflicting requests 2024-04-06T02:40:28.257 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.257 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 2: conflicting requests 2024-04-06T02:40:28.257 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.257 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 3: conflicting requests 2024-04-06T02:40:28.257 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.257 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 4: conflicting requests 2024-04-06T02:40:28.257 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.257 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 5: conflicting requests 2024-04-06T02:40:28.257 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.258 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 6: conflicting requests 2024-04-06T02:40:28.258 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.258 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 7: conflicting requests 2024-04-06T02:40:28.258 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.258 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 8: conflicting requests 2024-04-06T02:40:28.258 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.258 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 9: conflicting requests 2024-04-06T02:40:28.258 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.258 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 10: conflicting requests 2024-04-06T02:40:28.258 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.258 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 11: conflicting requests 2024-04-06T02:40:28.258 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.258 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 12: conflicting requests 2024-04-06T02:40:28.259 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.259 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 13: conflicting requests 2024-04-06T02:40:28.259 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.259 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 14: conflicting requests 2024-04-06T02:40:28.259 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.259 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 15: conflicting requests 2024-04-06T02:40:28.259 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.259 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 16: conflicting requests 2024-04-06T02:40:28.259 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.259 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 17: conflicting requests 2024-04-06T02:40:28.259 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.259 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 18: conflicting requests 2024-04-06T02:40:28.260 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.260 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 19: conflicting requests 2024-04-06T02:40:28.260 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.260 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 20: conflicting requests 2024-04-06T02:40:28.260 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.260 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 21: conflicting requests 2024-04-06T02:40:28.260 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.260 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 22: conflicting requests 2024-04-06T02:40:28.260 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.260 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 23: conflicting requests 2024-04-06T02:40:28.260 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.260 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 24: conflicting requests 2024-04-06T02:40:28.260 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.261 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 25: conflicting requests 2024-04-06T02:40:28.261 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.261 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 26: conflicting requests 2024-04-06T02:40:28.261 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.261 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 27: conflicting requests 2024-04-06T02:40:28.261 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.261 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 28: conflicting requests 2024-04-06T02:40:28.261 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.261 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 29: conflicting requests 2024-04-06T02:40:28.261 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.261 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 30: conflicting requests 2024-04-06T02:40:28.261 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.262 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 31: conflicting requests 2024-04-06T02:40:28.262 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.262 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 32: conflicting requests 2024-04-06T02:40:28.262 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.262 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 33: conflicting requests 2024-04-06T02:40:28.262 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.262 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 34: conflicting requests 2024-04-06T02:40:28.262 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.262 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 35: conflicting requests 2024-04-06T02:40:28.262 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.262 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 36: conflicting requests 2024-04-06T02:40:28.262 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.262 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 37: conflicting requests 2024-04-06T02:40:28.262 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.263 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.263 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.263 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:40:28.263 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.263 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.263 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.263 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.263 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 39: conflicting requests 2024-04-06T02:40:28.263 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.263 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.263 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.263 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.263 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.263 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.263 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:40:28.264 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.264 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.264 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.264 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.264 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:40:28.264 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.264 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.264 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.264 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.264 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:40:28.264 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.264 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.264 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.265 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.265 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:40:28.265 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.265 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.265 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.265 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.265 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:40:28.265 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.265 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.265 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.265 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.265 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:40:28.265 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.265 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.266 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.266 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:40:28.325 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-04-06T02:40:28.325 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:40:28.326 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-04-06T02:40:28.326 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:40:28.326 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-04-06T02:40:28.326 INFO:teuthology.orchestra.run.smithi184.stdout: librados-devel x86_64 2:18.2.1-0.el9 ceph 127 k 2024-04-06T02:40:28.326 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:28.326 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-04-06T02:40:28.326 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:40:28.326 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-04-06T02:40:28.326 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:28.327 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 127 k 2024-04-06T02:40:28.327 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 456 k 2024-04-06T02:40:28.327 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-04-06T02:40:28.390 INFO:teuthology.orchestra.run.smithi184.stdout:librados-devel-18.2.1-0.el9.x86_64.rpm 2.0 MB/s | 127 kB 00:00 2024-04-06T02:40:28.391 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:40:28.391 INFO:teuthology.orchestra.run.smithi184.stdout:Total 1.9 MB/s | 127 kB 00:00 2024-04-06T02:40:28.391 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-04-06T02:40:28.397 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-04-06T02:40:28.397 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-04-06T02:40:28.458 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-04-06T02:40:28.458 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-04-06T02:40:28.633 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-04-06T02:40:28.722 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:29.155 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:29.451 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:29.451 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:29.452 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-04-06T02:40:29.452 INFO:teuthology.orchestra.run.smithi184.stdout: librados-devel-2:18.2.1-0.el9.x86_64 2024-04-06T02:40:29.452 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:29.452 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-04-06T02:40:29.696 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install libcephfs2 2024-04-06T02:40:30.302 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:52 ago on Sat 06 Apr 2024 02:38:38 AM UTC. 2024-04-06T02:40:30.598 INFO:teuthology.orchestra.run.smithi184.stderr:Modular dependency problems: 2024-04-06T02:40:30.598 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-04-06T02:40:30.599 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 1: conflicting requests 2024-04-06T02:40:30.599 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.599 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 2: conflicting requests 2024-04-06T02:40:30.599 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.599 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 3: conflicting requests 2024-04-06T02:40:30.599 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.599 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 4: conflicting requests 2024-04-06T02:40:30.599 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.599 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 5: conflicting requests 2024-04-06T02:40:30.599 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.599 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 6: conflicting requests 2024-04-06T02:40:30.599 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.599 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 7: conflicting requests 2024-04-06T02:40:30.599 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.600 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 8: conflicting requests 2024-04-06T02:40:30.600 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.600 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 9: conflicting requests 2024-04-06T02:40:30.600 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.600 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 10: conflicting requests 2024-04-06T02:40:30.600 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.600 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 11: conflicting requests 2024-04-06T02:40:30.600 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.600 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 12: conflicting requests 2024-04-06T02:40:30.600 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.600 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 13: conflicting requests 2024-04-06T02:40:30.600 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.600 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 14: conflicting requests 2024-04-06T02:40:30.600 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.601 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 15: conflicting requests 2024-04-06T02:40:30.601 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.601 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 16: conflicting requests 2024-04-06T02:40:30.601 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.601 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 17: conflicting requests 2024-04-06T02:40:30.601 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.601 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 18: conflicting requests 2024-04-06T02:40:30.601 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.601 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 19: conflicting requests 2024-04-06T02:40:30.601 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.601 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 20: conflicting requests 2024-04-06T02:40:30.601 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.601 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 21: conflicting requests 2024-04-06T02:40:30.602 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.602 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 22: conflicting requests 2024-04-06T02:40:30.602 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.602 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 23: conflicting requests 2024-04-06T02:40:30.602 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.602 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 24: conflicting requests 2024-04-06T02:40:30.602 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.602 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 25: conflicting requests 2024-04-06T02:40:30.602 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.602 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 26: conflicting requests 2024-04-06T02:40:30.602 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.602 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 27: conflicting requests 2024-04-06T02:40:30.602 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.603 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 28: conflicting requests 2024-04-06T02:40:30.603 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.603 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 29: conflicting requests 2024-04-06T02:40:30.603 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.603 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 30: conflicting requests 2024-04-06T02:40:30.603 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.603 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 31: conflicting requests 2024-04-06T02:40:30.603 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.603 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 32: conflicting requests 2024-04-06T02:40:30.603 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.603 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 33: conflicting requests 2024-04-06T02:40:30.603 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.603 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 34: conflicting requests 2024-04-06T02:40:30.604 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.604 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 35: conflicting requests 2024-04-06T02:40:30.604 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.604 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 36: conflicting requests 2024-04-06T02:40:30.604 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.604 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 37: conflicting requests 2024-04-06T02:40:30.604 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.604 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.604 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.604 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:40:30.604 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.604 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.604 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.604 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.605 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 39: conflicting requests 2024-04-06T02:40:30.605 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.605 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.605 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.605 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.605 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.605 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.605 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:40:30.605 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.605 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.605 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.605 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.605 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:40:30.605 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.605 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.606 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.606 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.606 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:40:30.606 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.606 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.606 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.606 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.606 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:40:30.606 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.606 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.606 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.606 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.606 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:40:30.606 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.607 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.607 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.607 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.607 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:40:30.607 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.607 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.607 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.607 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:40:30.607 INFO:teuthology.orchestra.run.smithi184.stdout:Package libcephfs2-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-06T02:40:30.668 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-04-06T02:40:30.669 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-04-06T02:40:30.669 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-04-06T02:40:30.740 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install libcephfs-devel 2024-04-06T02:40:31.351 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:53 ago on Sat 06 Apr 2024 02:38:38 AM UTC. 2024-04-06T02:40:31.642 INFO:teuthology.orchestra.run.smithi184.stderr:Modular dependency problems: 2024-04-06T02:40:31.643 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-04-06T02:40:31.643 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 1: conflicting requests 2024-04-06T02:40:31.643 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.643 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 2: conflicting requests 2024-04-06T02:40:31.643 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.643 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 3: conflicting requests 2024-04-06T02:40:31.643 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.643 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 4: conflicting requests 2024-04-06T02:40:31.643 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.643 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 5: conflicting requests 2024-04-06T02:40:31.643 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.644 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 6: conflicting requests 2024-04-06T02:40:31.644 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.644 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 7: conflicting requests 2024-04-06T02:40:31.644 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.644 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 8: conflicting requests 2024-04-06T02:40:31.644 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.644 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 9: conflicting requests 2024-04-06T02:40:31.644 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.644 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 10: conflicting requests 2024-04-06T02:40:31.644 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.644 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 11: conflicting requests 2024-04-06T02:40:31.644 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.644 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 12: conflicting requests 2024-04-06T02:40:31.645 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.645 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 13: conflicting requests 2024-04-06T02:40:31.645 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.645 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 14: conflicting requests 2024-04-06T02:40:31.645 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.645 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 15: conflicting requests 2024-04-06T02:40:31.645 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.645 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 16: conflicting requests 2024-04-06T02:40:31.645 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.645 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 17: conflicting requests 2024-04-06T02:40:31.645 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.645 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 18: conflicting requests 2024-04-06T02:40:31.645 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.646 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 19: conflicting requests 2024-04-06T02:40:31.646 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.646 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 20: conflicting requests 2024-04-06T02:40:31.646 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.646 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 21: conflicting requests 2024-04-06T02:40:31.646 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.646 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 22: conflicting requests 2024-04-06T02:40:31.646 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.646 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 23: conflicting requests 2024-04-06T02:40:31.646 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.646 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 24: conflicting requests 2024-04-06T02:40:31.646 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.646 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 25: conflicting requests 2024-04-06T02:40:31.647 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.647 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 26: conflicting requests 2024-04-06T02:40:31.647 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.647 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 27: conflicting requests 2024-04-06T02:40:31.647 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.647 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 28: conflicting requests 2024-04-06T02:40:31.647 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.647 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 29: conflicting requests 2024-04-06T02:40:31.647 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.647 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 30: conflicting requests 2024-04-06T02:40:31.647 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.647 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 31: conflicting requests 2024-04-06T02:40:31.647 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.648 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 32: conflicting requests 2024-04-06T02:40:31.648 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.648 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 33: conflicting requests 2024-04-06T02:40:31.648 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.648 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 34: conflicting requests 2024-04-06T02:40:31.648 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.648 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 35: conflicting requests 2024-04-06T02:40:31.648 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.648 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 36: conflicting requests 2024-04-06T02:40:31.648 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.648 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 37: conflicting requests 2024-04-06T02:40:31.648 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.648 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.648 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.649 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:40:31.649 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.649 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.649 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.649 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.649 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 39: conflicting requests 2024-04-06T02:40:31.649 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.649 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.649 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.649 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.649 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.649 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.649 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:40:31.649 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.650 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.650 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.650 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.650 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:40:31.650 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.650 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.650 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.650 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.650 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:40:31.650 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.650 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.650 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.650 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.651 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:40:31.651 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.651 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.651 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.651 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.651 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:40:31.651 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.651 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.651 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.651 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.652 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:40:31.652 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.652 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.652 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.652 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:40:31.712 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-04-06T02:40:31.712 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:40:31.713 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-04-06T02:40:31.713 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:40:31.713 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-04-06T02:40:31.713 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs-devel x86_64 2:18.2.1-0.el9 ceph 31 k 2024-04-06T02:40:31.713 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:31.713 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-04-06T02:40:31.713 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:40:31.713 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-04-06T02:40:31.713 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:31.713 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 31 k 2024-04-06T02:40:31.713 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 139 k 2024-04-06T02:40:31.713 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-04-06T02:40:31.779 INFO:teuthology.orchestra.run.smithi184.stdout:libcephfs-devel-18.2.1-0.el9.x86_64.rpm 472 kB/s | 31 kB 00:00 2024-04-06T02:40:31.779 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:40:31.780 INFO:teuthology.orchestra.run.smithi184.stdout:Total 463 kB/s | 31 kB 00:00 2024-04-06T02:40:31.780 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-04-06T02:40:31.783 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-04-06T02:40:31.783 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-04-06T02:40:31.812 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-04-06T02:40:31.812 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-04-06T02:40:31.895 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-04-06T02:40:32.019 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:32.396 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:32.666 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:32.666 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:32.666 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-04-06T02:40:32.666 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs-devel-2:18.2.1-0.el9.x86_64 2024-04-06T02:40:32.666 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:32.666 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-04-06T02:40:32.833 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install librados2 2024-04-06T02:40:33.434 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:55 ago on Sat 06 Apr 2024 02:38:38 AM UTC. 2024-04-06T02:40:33.731 INFO:teuthology.orchestra.run.smithi184.stderr:Modular dependency problems: 2024-04-06T02:40:33.731 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-04-06T02:40:33.731 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 1: conflicting requests 2024-04-06T02:40:33.731 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.731 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 2: conflicting requests 2024-04-06T02:40:33.731 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.731 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 3: conflicting requests 2024-04-06T02:40:33.731 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.731 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 4: conflicting requests 2024-04-06T02:40:33.731 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.732 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 5: conflicting requests 2024-04-06T02:40:33.732 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.732 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 6: conflicting requests 2024-04-06T02:40:33.732 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.732 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 7: conflicting requests 2024-04-06T02:40:33.732 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.732 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 8: conflicting requests 2024-04-06T02:40:33.732 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.732 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 9: conflicting requests 2024-04-06T02:40:33.732 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.732 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 10: conflicting requests 2024-04-06T02:40:33.732 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.732 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 11: conflicting requests 2024-04-06T02:40:33.733 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.733 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 12: conflicting requests 2024-04-06T02:40:33.733 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.733 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 13: conflicting requests 2024-04-06T02:40:33.733 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.733 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 14: conflicting requests 2024-04-06T02:40:33.733 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.733 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 15: conflicting requests 2024-04-06T02:40:33.733 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.733 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 16: conflicting requests 2024-04-06T02:40:33.733 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.733 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 17: conflicting requests 2024-04-06T02:40:33.733 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.734 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 18: conflicting requests 2024-04-06T02:40:33.734 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.734 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 19: conflicting requests 2024-04-06T02:40:33.734 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.734 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 20: conflicting requests 2024-04-06T02:40:33.734 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.734 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 21: conflicting requests 2024-04-06T02:40:33.734 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.734 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 22: conflicting requests 2024-04-06T02:40:33.734 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.734 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 23: conflicting requests 2024-04-06T02:40:33.734 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.734 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 24: conflicting requests 2024-04-06T02:40:33.735 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.735 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 25: conflicting requests 2024-04-06T02:40:33.735 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.735 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 26: conflicting requests 2024-04-06T02:40:33.735 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.735 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 27: conflicting requests 2024-04-06T02:40:33.735 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.735 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 28: conflicting requests 2024-04-06T02:40:33.735 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.735 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 29: conflicting requests 2024-04-06T02:40:33.735 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.735 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 30: conflicting requests 2024-04-06T02:40:33.735 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.736 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 31: conflicting requests 2024-04-06T02:40:33.736 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.736 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 32: conflicting requests 2024-04-06T02:40:33.736 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.736 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 33: conflicting requests 2024-04-06T02:40:33.736 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.736 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 34: conflicting requests 2024-04-06T02:40:33.736 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.736 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 35: conflicting requests 2024-04-06T02:40:33.736 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.736 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 36: conflicting requests 2024-04-06T02:40:33.736 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.736 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 37: conflicting requests 2024-04-06T02:40:33.737 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.737 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.737 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.737 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:40:33.737 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.737 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.737 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.737 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.737 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 39: conflicting requests 2024-04-06T02:40:33.737 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.737 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.737 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.737 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.737 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.738 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.738 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:40:33.738 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.738 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.738 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.738 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.738 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:40:33.738 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.738 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.738 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.738 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.738 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:40:33.739 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.739 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.739 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.739 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.739 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:40:33.739 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.739 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.739 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.739 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.739 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:40:33.739 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.739 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.739 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.740 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.740 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:40:33.740 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.740 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.740 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.740 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:40:33.740 INFO:teuthology.orchestra.run.smithi184.stdout:Package librados2-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-06T02:40:33.799 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-04-06T02:40:33.800 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-04-06T02:40:33.800 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-04-06T02:40:33.866 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install librbd1 2024-04-06T02:40:34.466 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:56 ago on Sat 06 Apr 2024 02:38:38 AM UTC. 2024-04-06T02:40:34.753 INFO:teuthology.orchestra.run.smithi184.stderr:Modular dependency problems: 2024-04-06T02:40:34.753 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-04-06T02:40:34.753 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 1: conflicting requests 2024-04-06T02:40:34.753 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.753 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 2: conflicting requests 2024-04-06T02:40:34.753 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.753 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 3: conflicting requests 2024-04-06T02:40:34.753 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.754 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 4: conflicting requests 2024-04-06T02:40:34.754 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.754 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 5: conflicting requests 2024-04-06T02:40:34.754 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.754 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 6: conflicting requests 2024-04-06T02:40:34.754 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.754 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 7: conflicting requests 2024-04-06T02:40:34.754 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.754 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 8: conflicting requests 2024-04-06T02:40:34.754 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.754 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 9: conflicting requests 2024-04-06T02:40:34.754 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.755 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 10: conflicting requests 2024-04-06T02:40:34.755 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.755 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 11: conflicting requests 2024-04-06T02:40:34.755 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.755 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 12: conflicting requests 2024-04-06T02:40:34.755 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.755 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 13: conflicting requests 2024-04-06T02:40:34.755 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.755 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 14: conflicting requests 2024-04-06T02:40:34.755 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.755 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 15: conflicting requests 2024-04-06T02:40:34.755 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.755 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 16: conflicting requests 2024-04-06T02:40:34.755 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.756 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 17: conflicting requests 2024-04-06T02:40:34.756 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.756 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 18: conflicting requests 2024-04-06T02:40:34.756 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.756 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 19: conflicting requests 2024-04-06T02:40:34.756 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.756 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 20: conflicting requests 2024-04-06T02:40:34.756 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.756 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 21: conflicting requests 2024-04-06T02:40:34.756 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.756 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 22: conflicting requests 2024-04-06T02:40:34.756 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.756 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 23: conflicting requests 2024-04-06T02:40:34.756 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.757 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 24: conflicting requests 2024-04-06T02:40:34.757 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.757 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 25: conflicting requests 2024-04-06T02:40:34.757 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.757 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 26: conflicting requests 2024-04-06T02:40:34.757 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.757 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 27: conflicting requests 2024-04-06T02:40:34.757 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.757 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 28: conflicting requests 2024-04-06T02:40:34.757 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.757 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 29: conflicting requests 2024-04-06T02:40:34.757 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.758 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 30: conflicting requests 2024-04-06T02:40:34.758 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.758 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 31: conflicting requests 2024-04-06T02:40:34.758 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.758 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 32: conflicting requests 2024-04-06T02:40:34.758 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.758 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 33: conflicting requests 2024-04-06T02:40:34.758 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.758 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 34: conflicting requests 2024-04-06T02:40:34.758 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.759 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 35: conflicting requests 2024-04-06T02:40:34.759 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.759 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 36: conflicting requests 2024-04-06T02:40:34.759 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.759 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 37: conflicting requests 2024-04-06T02:40:34.759 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.759 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.759 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.759 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:40:34.759 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.759 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.759 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.759 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.760 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 39: conflicting requests 2024-04-06T02:40:34.760 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.760 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.760 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.760 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.760 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.760 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.760 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:40:34.760 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.760 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.760 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.760 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.760 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:40:34.760 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.761 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.761 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.761 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.761 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:40:34.761 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.761 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.761 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.761 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.761 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:40:34.761 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.761 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.761 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.762 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.762 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:40:34.762 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.762 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.762 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.762 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.762 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:40:34.762 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.762 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.762 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.762 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:40:34.763 INFO:teuthology.orchestra.run.smithi184.stdout:Package librbd1-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-06T02:40:34.821 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-04-06T02:40:34.822 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-04-06T02:40:34.822 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-04-06T02:40:34.889 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-rados 2024-04-06T02:40:35.488 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:57 ago on Sat 06 Apr 2024 02:38:38 AM UTC. 2024-04-06T02:40:35.780 INFO:teuthology.orchestra.run.smithi184.stderr:Modular dependency problems: 2024-04-06T02:40:35.780 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-04-06T02:40:35.780 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 1: conflicting requests 2024-04-06T02:40:35.780 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.780 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 2: conflicting requests 2024-04-06T02:40:35.780 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.780 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 3: conflicting requests 2024-04-06T02:40:35.780 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.780 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 4: conflicting requests 2024-04-06T02:40:35.780 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.781 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 5: conflicting requests 2024-04-06T02:40:35.781 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.781 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 6: conflicting requests 2024-04-06T02:40:35.781 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.781 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 7: conflicting requests 2024-04-06T02:40:35.781 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.781 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 8: conflicting requests 2024-04-06T02:40:35.781 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.781 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 9: conflicting requests 2024-04-06T02:40:35.781 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.781 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 10: conflicting requests 2024-04-06T02:40:35.781 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.781 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 11: conflicting requests 2024-04-06T02:40:35.781 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.782 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 12: conflicting requests 2024-04-06T02:40:35.782 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.782 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 13: conflicting requests 2024-04-06T02:40:35.782 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.782 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 14: conflicting requests 2024-04-06T02:40:35.782 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.782 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 15: conflicting requests 2024-04-06T02:40:35.782 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.782 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 16: conflicting requests 2024-04-06T02:40:35.782 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.782 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 17: conflicting requests 2024-04-06T02:40:35.782 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.783 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 18: conflicting requests 2024-04-06T02:40:35.783 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.783 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 19: conflicting requests 2024-04-06T02:40:35.783 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.783 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 20: conflicting requests 2024-04-06T02:40:35.783 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.783 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 21: conflicting requests 2024-04-06T02:40:35.783 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.783 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 22: conflicting requests 2024-04-06T02:40:35.783 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.783 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 23: conflicting requests 2024-04-06T02:40:35.783 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.783 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 24: conflicting requests 2024-04-06T02:40:35.783 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.784 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 25: conflicting requests 2024-04-06T02:40:35.784 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.784 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 26: conflicting requests 2024-04-06T02:40:35.784 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.784 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 27: conflicting requests 2024-04-06T02:40:35.784 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.784 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 28: conflicting requests 2024-04-06T02:40:35.784 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.784 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 29: conflicting requests 2024-04-06T02:40:35.784 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.784 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 30: conflicting requests 2024-04-06T02:40:35.784 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.784 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 31: conflicting requests 2024-04-06T02:40:35.784 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.785 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 32: conflicting requests 2024-04-06T02:40:35.785 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.785 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 33: conflicting requests 2024-04-06T02:40:35.785 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.785 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 34: conflicting requests 2024-04-06T02:40:35.785 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.785 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 35: conflicting requests 2024-04-06T02:40:35.785 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.785 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 36: conflicting requests 2024-04-06T02:40:35.785 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.785 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 37: conflicting requests 2024-04-06T02:40:35.785 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.785 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.785 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.786 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:40:35.786 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.786 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.786 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.786 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.786 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 39: conflicting requests 2024-04-06T02:40:35.786 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.786 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.786 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.786 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.786 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.786 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.786 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:40:35.786 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.786 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.787 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.787 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.787 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:40:35.787 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.787 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.787 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.787 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.787 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:40:35.787 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.787 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.787 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.787 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.788 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:40:35.788 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.788 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.788 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.788 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.788 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:40:35.788 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.788 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.788 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.788 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.788 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:40:35.788 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.788 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.788 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.789 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:40:35.789 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-rados-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-06T02:40:35.850 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-04-06T02:40:35.851 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-04-06T02:40:35.852 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-04-06T02:40:35.922 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-rgw 2024-04-06T02:40:36.525 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:58 ago on Sat 06 Apr 2024 02:38:38 AM UTC. 2024-04-06T02:40:36.819 INFO:teuthology.orchestra.run.smithi184.stderr:Modular dependency problems: 2024-04-06T02:40:36.819 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-04-06T02:40:36.819 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 1: conflicting requests 2024-04-06T02:40:36.819 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.819 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 2: conflicting requests 2024-04-06T02:40:36.819 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.819 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 3: conflicting requests 2024-04-06T02:40:36.819 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.819 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 4: conflicting requests 2024-04-06T02:40:36.819 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.819 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 5: conflicting requests 2024-04-06T02:40:36.819 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.820 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 6: conflicting requests 2024-04-06T02:40:36.820 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.820 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 7: conflicting requests 2024-04-06T02:40:36.820 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.820 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 8: conflicting requests 2024-04-06T02:40:36.820 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.820 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 9: conflicting requests 2024-04-06T02:40:36.820 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.820 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 10: conflicting requests 2024-04-06T02:40:36.820 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.820 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 11: conflicting requests 2024-04-06T02:40:36.820 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.820 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 12: conflicting requests 2024-04-06T02:40:36.821 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.821 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 13: conflicting requests 2024-04-06T02:40:36.821 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.821 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 14: conflicting requests 2024-04-06T02:40:36.821 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.821 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 15: conflicting requests 2024-04-06T02:40:36.821 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.821 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 16: conflicting requests 2024-04-06T02:40:36.821 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.821 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 17: conflicting requests 2024-04-06T02:40:36.821 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.821 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 18: conflicting requests 2024-04-06T02:40:36.821 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.822 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 19: conflicting requests 2024-04-06T02:40:36.822 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.822 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 20: conflicting requests 2024-04-06T02:40:36.822 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.822 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 21: conflicting requests 2024-04-06T02:40:36.822 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.822 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 22: conflicting requests 2024-04-06T02:40:36.822 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.822 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 23: conflicting requests 2024-04-06T02:40:36.822 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.822 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 24: conflicting requests 2024-04-06T02:40:36.822 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.822 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 25: conflicting requests 2024-04-06T02:40:36.823 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.823 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 26: conflicting requests 2024-04-06T02:40:36.823 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.823 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 27: conflicting requests 2024-04-06T02:40:36.823 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.823 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 28: conflicting requests 2024-04-06T02:40:36.823 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.823 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 29: conflicting requests 2024-04-06T02:40:36.823 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.823 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 30: conflicting requests 2024-04-06T02:40:36.823 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.823 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 31: conflicting requests 2024-04-06T02:40:36.823 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 32: conflicting requests 2024-04-06T02:40:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 33: conflicting requests 2024-04-06T02:40:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 34: conflicting requests 2024-04-06T02:40:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 35: conflicting requests 2024-04-06T02:40:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 36: conflicting requests 2024-04-06T02:40:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 37: conflicting requests 2024-04-06T02:40:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.824 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.825 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:40:36.825 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.825 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.825 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.825 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.825 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 39: conflicting requests 2024-04-06T02:40:36.825 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.825 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.825 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.825 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.825 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.825 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.825 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:40:36.826 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.826 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.826 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.826 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.826 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:40:36.826 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.826 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.826 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.826 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.826 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:40:36.826 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.826 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.826 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.827 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.827 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:40:36.827 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.827 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.827 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.827 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.827 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:40:36.827 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.827 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.827 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.827 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.827 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:40:36.827 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.828 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.828 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.828 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:40:36.828 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-rgw-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-06T02:40:36.888 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-04-06T02:40:36.889 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-04-06T02:40:36.889 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-04-06T02:40:36.956 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-cephfs 2024-04-06T02:40:37.553 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:59 ago on Sat 06 Apr 2024 02:38:38 AM UTC. 2024-04-06T02:40:37.845 INFO:teuthology.orchestra.run.smithi184.stderr:Modular dependency problems: 2024-04-06T02:40:37.845 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-04-06T02:40:37.845 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 1: conflicting requests 2024-04-06T02:40:37.845 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.845 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 2: conflicting requests 2024-04-06T02:40:37.845 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.846 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 3: conflicting requests 2024-04-06T02:40:37.846 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.846 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 4: conflicting requests 2024-04-06T02:40:37.846 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.846 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 5: conflicting requests 2024-04-06T02:40:37.846 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.846 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 6: conflicting requests 2024-04-06T02:40:37.846 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.846 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 7: conflicting requests 2024-04-06T02:40:37.846 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.846 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 8: conflicting requests 2024-04-06T02:40:37.846 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.847 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 9: conflicting requests 2024-04-06T02:40:37.847 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.847 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 10: conflicting requests 2024-04-06T02:40:37.847 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.847 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 11: conflicting requests 2024-04-06T02:40:37.847 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.847 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 12: conflicting requests 2024-04-06T02:40:37.847 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.847 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 13: conflicting requests 2024-04-06T02:40:37.847 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.847 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 14: conflicting requests 2024-04-06T02:40:37.847 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.847 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 15: conflicting requests 2024-04-06T02:40:37.848 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.848 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 16: conflicting requests 2024-04-06T02:40:37.848 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.848 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 17: conflicting requests 2024-04-06T02:40:37.848 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.848 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 18: conflicting requests 2024-04-06T02:40:37.848 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.848 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 19: conflicting requests 2024-04-06T02:40:37.848 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.848 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 20: conflicting requests 2024-04-06T02:40:37.848 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.848 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 21: conflicting requests 2024-04-06T02:40:37.848 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.849 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 22: conflicting requests 2024-04-06T02:40:37.849 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.849 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 23: conflicting requests 2024-04-06T02:40:37.849 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.849 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 24: conflicting requests 2024-04-06T02:40:37.849 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.849 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 25: conflicting requests 2024-04-06T02:40:37.849 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.849 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 26: conflicting requests 2024-04-06T02:40:37.849 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.849 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 27: conflicting requests 2024-04-06T02:40:37.849 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.849 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 28: conflicting requests 2024-04-06T02:40:37.850 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.850 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 29: conflicting requests 2024-04-06T02:40:37.850 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.850 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 30: conflicting requests 2024-04-06T02:40:37.850 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.850 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 31: conflicting requests 2024-04-06T02:40:37.850 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.850 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 32: conflicting requests 2024-04-06T02:40:37.850 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.850 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 33: conflicting requests 2024-04-06T02:40:37.850 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.850 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 34: conflicting requests 2024-04-06T02:40:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 35: conflicting requests 2024-04-06T02:40:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 36: conflicting requests 2024-04-06T02:40:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 37: conflicting requests 2024-04-06T02:40:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:40:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.851 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.852 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 39: conflicting requests 2024-04-06T02:40:37.852 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.852 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.852 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.852 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.852 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.852 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.852 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:40:37.852 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.852 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.852 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.852 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.852 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:40:37.853 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.853 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.853 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.853 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.853 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:40:37.853 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.853 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.853 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.853 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.853 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:40:37.853 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.853 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.853 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.854 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.854 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:40:37.854 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.854 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.854 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.854 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.854 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:40:37.854 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.854 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.854 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.854 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:40:37.855 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-cephfs-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-06T02:40:37.915 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-04-06T02:40:37.916 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-04-06T02:40:37.916 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-04-06T02:40:37.982 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-rbd 2024-04-06T02:40:38.574 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:02:00 ago on Sat 06 Apr 2024 02:38:38 AM UTC. 2024-04-06T02:40:38.860 INFO:teuthology.orchestra.run.smithi184.stderr:Modular dependency problems: 2024-04-06T02:40:38.860 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-04-06T02:40:38.861 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 1: conflicting requests 2024-04-06T02:40:38.861 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.861 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 2: conflicting requests 2024-04-06T02:40:38.861 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.861 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 3: conflicting requests 2024-04-06T02:40:38.861 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.861 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 4: conflicting requests 2024-04-06T02:40:38.861 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.861 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 5: conflicting requests 2024-04-06T02:40:38.861 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.861 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 6: conflicting requests 2024-04-06T02:40:38.861 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.862 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 7: conflicting requests 2024-04-06T02:40:38.862 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.862 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 8: conflicting requests 2024-04-06T02:40:38.862 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.862 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 9: conflicting requests 2024-04-06T02:40:38.862 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.862 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 10: conflicting requests 2024-04-06T02:40:38.862 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.862 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 11: conflicting requests 2024-04-06T02:40:38.862 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.862 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 12: conflicting requests 2024-04-06T02:40:38.862 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.863 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 13: conflicting requests 2024-04-06T02:40:38.863 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.863 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 14: conflicting requests 2024-04-06T02:40:38.863 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.863 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 15: conflicting requests 2024-04-06T02:40:38.863 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.863 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 16: conflicting requests 2024-04-06T02:40:38.863 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.863 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 17: conflicting requests 2024-04-06T02:40:38.863 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.863 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 18: conflicting requests 2024-04-06T02:40:38.863 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.864 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 19: conflicting requests 2024-04-06T02:40:38.864 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.864 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 20: conflicting requests 2024-04-06T02:40:38.864 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.864 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 21: conflicting requests 2024-04-06T02:40:38.864 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.864 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 22: conflicting requests 2024-04-06T02:40:38.864 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.864 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 23: conflicting requests 2024-04-06T02:40:38.864 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.865 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 24: conflicting requests 2024-04-06T02:40:38.865 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.865 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 25: conflicting requests 2024-04-06T02:40:38.865 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.865 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 26: conflicting requests 2024-04-06T02:40:38.865 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.865 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 27: conflicting requests 2024-04-06T02:40:38.865 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.865 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 28: conflicting requests 2024-04-06T02:40:38.865 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.865 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 29: conflicting requests 2024-04-06T02:40:38.865 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.865 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 30: conflicting requests 2024-04-06T02:40:38.865 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.866 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 31: conflicting requests 2024-04-06T02:40:38.866 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.866 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 32: conflicting requests 2024-04-06T02:40:38.866 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.866 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 33: conflicting requests 2024-04-06T02:40:38.866 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.866 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 34: conflicting requests 2024-04-06T02:40:38.866 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.866 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 35: conflicting requests 2024-04-06T02:40:38.866 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.866 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 36: conflicting requests 2024-04-06T02:40:38.867 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.867 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 37: conflicting requests 2024-04-06T02:40:38.867 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.867 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.867 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.867 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:40:38.867 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.867 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.867 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.867 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.867 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 39: conflicting requests 2024-04-06T02:40:38.867 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.867 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.868 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.868 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.868 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.868 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.868 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:40:38.868 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.868 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.868 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.868 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.868 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:40:38.868 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.868 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.868 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.868 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.869 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:40:38.869 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.869 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.869 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.869 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.869 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:40:38.869 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.869 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.869 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.869 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.869 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:40:38.869 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.869 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.870 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.870 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.870 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:40:38.870 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.870 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.870 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.870 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:40:38.870 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-rbd-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-06T02:40:38.930 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-04-06T02:40:38.931 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-04-06T02:40:38.931 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-04-06T02:40:39.001 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install rbd-fuse 2024-04-06T02:40:39.599 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:02:01 ago on Sat 06 Apr 2024 02:38:38 AM UTC. 2024-04-06T02:40:39.888 INFO:teuthology.orchestra.run.smithi184.stderr:Modular dependency problems: 2024-04-06T02:40:39.888 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-04-06T02:40:39.889 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 1: conflicting requests 2024-04-06T02:40:39.889 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.889 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 2: conflicting requests 2024-04-06T02:40:39.889 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.889 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 3: conflicting requests 2024-04-06T02:40:39.889 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.889 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 4: conflicting requests 2024-04-06T02:40:39.889 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.889 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 5: conflicting requests 2024-04-06T02:40:39.889 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.889 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 6: conflicting requests 2024-04-06T02:40:39.889 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.890 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 7: conflicting requests 2024-04-06T02:40:39.890 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.890 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 8: conflicting requests 2024-04-06T02:40:39.890 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.890 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 9: conflicting requests 2024-04-06T02:40:39.890 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.890 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 10: conflicting requests 2024-04-06T02:40:39.890 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.890 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 11: conflicting requests 2024-04-06T02:40:39.890 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.890 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 12: conflicting requests 2024-04-06T02:40:39.890 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.891 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 13: conflicting requests 2024-04-06T02:40:39.891 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.891 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 14: conflicting requests 2024-04-06T02:40:39.891 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.891 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 15: conflicting requests 2024-04-06T02:40:39.891 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.891 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 16: conflicting requests 2024-04-06T02:40:39.891 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.891 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 17: conflicting requests 2024-04-06T02:40:39.891 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.891 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 18: conflicting requests 2024-04-06T02:40:39.891 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.892 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 19: conflicting requests 2024-04-06T02:40:39.892 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.892 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 20: conflicting requests 2024-04-06T02:40:39.892 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.892 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 21: conflicting requests 2024-04-06T02:40:39.892 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.892 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 22: conflicting requests 2024-04-06T02:40:39.892 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.892 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 23: conflicting requests 2024-04-06T02:40:39.892 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.892 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 24: conflicting requests 2024-04-06T02:40:39.892 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.892 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 25: conflicting requests 2024-04-06T02:40:39.893 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.893 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 26: conflicting requests 2024-04-06T02:40:39.893 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.893 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 27: conflicting requests 2024-04-06T02:40:39.893 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.893 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 28: conflicting requests 2024-04-06T02:40:39.893 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.893 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 29: conflicting requests 2024-04-06T02:40:39.893 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.893 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 30: conflicting requests 2024-04-06T02:40:39.893 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.893 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 31: conflicting requests 2024-04-06T02:40:39.893 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.893 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 32: conflicting requests 2024-04-06T02:40:39.894 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.894 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 33: conflicting requests 2024-04-06T02:40:39.894 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.894 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 34: conflicting requests 2024-04-06T02:40:39.894 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.894 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 35: conflicting requests 2024-04-06T02:40:39.894 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.894 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 36: conflicting requests 2024-04-06T02:40:39.894 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.894 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 37: conflicting requests 2024-04-06T02:40:39.895 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.895 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.895 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.895 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:40:39.895 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.895 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.895 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.895 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.895 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 39: conflicting requests 2024-04-06T02:40:39.895 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.896 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.896 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.896 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.896 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.896 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.896 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:40:39.896 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.896 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.896 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.896 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.896 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:40:39.896 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.896 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.897 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.897 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.897 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:40:39.897 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.897 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.897 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.897 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.897 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:40:39.897 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.897 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.897 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.897 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.897 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:40:39.898 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.898 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.898 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.898 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.898 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:40:39.898 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.898 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.898 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.898 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:40:39.958 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-04-06T02:40:39.959 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:40:39.959 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-04-06T02:40:39.959 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:40:39.959 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-04-06T02:40:39.959 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-fuse x86_64 2:18.2.1-0.el9 ceph 86 k 2024-04-06T02:40:39.959 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:39.959 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-04-06T02:40:39.959 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:40:39.959 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-04-06T02:40:39.959 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:39.959 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 86 k 2024-04-06T02:40:39.960 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 226 k 2024-04-06T02:40:39.960 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-04-06T02:40:40.024 INFO:teuthology.orchestra.run.smithi184.stdout:rbd-fuse-18.2.1-0.el9.x86_64.rpm 1.3 MB/s | 86 kB 00:00 2024-04-06T02:40:40.024 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:40:40.024 INFO:teuthology.orchestra.run.smithi184.stdout:Total 1.3 MB/s | 86 kB 00:00 2024-04-06T02:40:40.024 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-04-06T02:40:40.031 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-04-06T02:40:40.031 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-04-06T02:40:40.091 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-04-06T02:40:40.092 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-04-06T02:40:40.234 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-04-06T02:40:40.322 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:40.751 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:41.091 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:41.091 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:41.091 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-04-06T02:40:41.091 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-fuse-2:18.2.1-0.el9.x86_64 2024-04-06T02:40:41.091 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:41.092 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-04-06T02:40:41.311 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install rbd-mirror 2024-04-06T02:40:41.951 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:02:03 ago on Sat 06 Apr 2024 02:38:38 AM UTC. 2024-04-06T02:40:42.244 INFO:teuthology.orchestra.run.smithi184.stderr:Modular dependency problems: 2024-04-06T02:40:42.244 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-04-06T02:40:42.244 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 1: conflicting requests 2024-04-06T02:40:42.244 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.244 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 2: conflicting requests 2024-04-06T02:40:42.244 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.244 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 3: conflicting requests 2024-04-06T02:40:42.245 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.245 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 4: conflicting requests 2024-04-06T02:40:42.245 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.245 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 5: conflicting requests 2024-04-06T02:40:42.245 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.245 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 6: conflicting requests 2024-04-06T02:40:42.245 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.245 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 7: conflicting requests 2024-04-06T02:40:42.245 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.245 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 8: conflicting requests 2024-04-06T02:40:42.245 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.245 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 9: conflicting requests 2024-04-06T02:40:42.245 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.245 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 10: conflicting requests 2024-04-06T02:40:42.246 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.246 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 11: conflicting requests 2024-04-06T02:40:42.246 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.246 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 12: conflicting requests 2024-04-06T02:40:42.246 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.246 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 13: conflicting requests 2024-04-06T02:40:42.246 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.246 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 14: conflicting requests 2024-04-06T02:40:42.246 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.246 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 15: conflicting requests 2024-04-06T02:40:42.246 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.246 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 16: conflicting requests 2024-04-06T02:40:42.247 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.247 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 17: conflicting requests 2024-04-06T02:40:42.247 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.247 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 18: conflicting requests 2024-04-06T02:40:42.247 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.247 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 19: conflicting requests 2024-04-06T02:40:42.247 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.247 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 20: conflicting requests 2024-04-06T02:40:42.247 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.247 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 21: conflicting requests 2024-04-06T02:40:42.247 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.247 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 22: conflicting requests 2024-04-06T02:40:42.247 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.248 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 23: conflicting requests 2024-04-06T02:40:42.248 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.248 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 24: conflicting requests 2024-04-06T02:40:42.248 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.248 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 25: conflicting requests 2024-04-06T02:40:42.248 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.248 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 26: conflicting requests 2024-04-06T02:40:42.248 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.248 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 27: conflicting requests 2024-04-06T02:40:42.248 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.248 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 28: conflicting requests 2024-04-06T02:40:42.248 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.248 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 29: conflicting requests 2024-04-06T02:40:42.249 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.249 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 30: conflicting requests 2024-04-06T02:40:42.249 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.249 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 31: conflicting requests 2024-04-06T02:40:42.249 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.249 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 32: conflicting requests 2024-04-06T02:40:42.249 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.249 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 33: conflicting requests 2024-04-06T02:40:42.249 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.249 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 34: conflicting requests 2024-04-06T02:40:42.249 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.249 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 35: conflicting requests 2024-04-06T02:40:42.249 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.249 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 36: conflicting requests 2024-04-06T02:40:42.250 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.250 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 37: conflicting requests 2024-04-06T02:40:42.250 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.250 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.250 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.250 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:40:42.250 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.250 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.250 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.250 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.250 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 39: conflicting requests 2024-04-06T02:40:42.250 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.251 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.251 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.251 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.251 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.251 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.251 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:40:42.251 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.251 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.251 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.251 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.251 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:40:42.251 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.251 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.251 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.252 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.252 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:40:42.252 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.252 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.252 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.252 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.252 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:40:42.252 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.252 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.252 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.252 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.252 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:40:42.252 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.252 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.252 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.253 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.253 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:40:42.253 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.253 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.253 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.253 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:40:42.314 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-04-06T02:40:42.315 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:40:42.315 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-04-06T02:40:42.315 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:40:42.315 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-04-06T02:40:42.315 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-mirror x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-04-06T02:40:42.315 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:42.315 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-04-06T02:40:42.315 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:40:42.315 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-04-06T02:40:42.315 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:42.316 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 3.0 M 2024-04-06T02:40:42.316 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 12 M 2024-04-06T02:40:42.316 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-04-06T02:40:42.435 INFO:teuthology.orchestra.run.smithi184.stdout:rbd-mirror-18.2.1-0.el9.x86_64.rpm 25 MB/s | 3.0 MB 00:00 2024-04-06T02:40:42.436 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:40:42.437 INFO:teuthology.orchestra.run.smithi184.stdout:Total 25 MB/s | 3.0 MB 00:00 2024-04-06T02:40:42.437 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-04-06T02:40:42.446 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-04-06T02:40:42.446 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-04-06T02:40:42.524 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-04-06T02:40:42.524 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-04-06T02:40:42.940 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-04-06T02:40:43.003 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:43.028 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:43.028 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-06T02:40:43.029 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-06T02:40:43.029 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-06T02:40:43.029 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-06T02:40:43.029 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:44.155 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:44.155 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:44.155 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-04-06T02:40:44.155 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-mirror-2:18.2.1-0.el9.x86_64 2024-04-06T02:40:44.156 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:44.156 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-04-06T02:40:44.386 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install rbd-nbd 2024-04-06T02:40:44.996 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:02:07 ago on Sat 06 Apr 2024 02:38:38 AM UTC. 2024-04-06T02:40:45.285 INFO:teuthology.orchestra.run.smithi184.stderr:Modular dependency problems: 2024-04-06T02:40:45.285 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-04-06T02:40:45.286 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 1: conflicting requests 2024-04-06T02:40:45.286 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.286 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 2: conflicting requests 2024-04-06T02:40:45.286 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.286 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 3: conflicting requests 2024-04-06T02:40:45.286 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.286 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 4: conflicting requests 2024-04-06T02:40:45.286 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.286 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 5: conflicting requests 2024-04-06T02:40:45.286 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.286 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 6: conflicting requests 2024-04-06T02:40:45.286 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.287 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 7: conflicting requests 2024-04-06T02:40:45.287 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.287 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 8: conflicting requests 2024-04-06T02:40:45.287 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.287 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 9: conflicting requests 2024-04-06T02:40:45.287 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.287 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 10: conflicting requests 2024-04-06T02:40:45.287 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.287 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 11: conflicting requests 2024-04-06T02:40:45.287 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.287 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 12: conflicting requests 2024-04-06T02:40:45.287 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.287 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 13: conflicting requests 2024-04-06T02:40:45.288 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.288 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 14: conflicting requests 2024-04-06T02:40:45.288 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.288 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 15: conflicting requests 2024-04-06T02:40:45.288 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.288 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 16: conflicting requests 2024-04-06T02:40:45.288 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.288 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 17: conflicting requests 2024-04-06T02:40:45.288 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.288 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 18: conflicting requests 2024-04-06T02:40:45.288 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.288 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 19: conflicting requests 2024-04-06T02:40:45.289 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.289 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 20: conflicting requests 2024-04-06T02:40:45.289 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.289 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 21: conflicting requests 2024-04-06T02:40:45.289 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.289 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 22: conflicting requests 2024-04-06T02:40:45.289 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.289 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 23: conflicting requests 2024-04-06T02:40:45.289 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.289 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 24: conflicting requests 2024-04-06T02:40:45.289 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.289 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 25: conflicting requests 2024-04-06T02:40:45.289 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.289 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 26: conflicting requests 2024-04-06T02:40:45.290 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.290 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 27: conflicting requests 2024-04-06T02:40:45.290 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.290 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 28: conflicting requests 2024-04-06T02:40:45.290 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.290 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 29: conflicting requests 2024-04-06T02:40:45.290 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.290 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 30: conflicting requests 2024-04-06T02:40:45.290 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.290 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 31: conflicting requests 2024-04-06T02:40:45.290 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.290 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 32: conflicting requests 2024-04-06T02:40:45.290 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.291 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 33: conflicting requests 2024-04-06T02:40:45.291 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.291 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 34: conflicting requests 2024-04-06T02:40:45.291 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.291 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 35: conflicting requests 2024-04-06T02:40:45.291 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.291 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 36: conflicting requests 2024-04-06T02:40:45.291 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.291 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 37: conflicting requests 2024-04-06T02:40:45.291 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.291 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.291 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.291 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 38: cannot install the best candidate for the job 2024-04-06T02:40:45.291 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.291 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.292 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.292 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.292 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 39: conflicting requests 2024-04-06T02:40:45.292 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.292 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.292 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.292 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.292 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.292 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.292 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 40: cannot install the best candidate for the job 2024-04-06T02:40:45.292 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.292 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.292 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.293 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.293 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 41: cannot install the best candidate for the job 2024-04-06T02:40:45.293 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.293 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.293 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.293 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.293 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 42: cannot install the best candidate for the job 2024-04-06T02:40:45.293 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.293 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.293 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.293 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.293 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 43: cannot install the best candidate for the job 2024-04-06T02:40:45.293 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.293 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.294 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.294 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.294 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 44: cannot install the best candidate for the job 2024-04-06T02:40:45.294 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.294 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.294 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.294 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.294 INFO:teuthology.orchestra.run.smithi184.stderr: Problem 45: cannot install the best candidate for the job 2024-04-06T02:40:45.294 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.294 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.294 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.294 INFO:teuthology.orchestra.run.smithi184.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-06T02:40:45.353 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-04-06T02:40:45.354 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:40:45.354 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-04-06T02:40:45.354 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:40:45.354 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-04-06T02:40:45.354 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-nbd x86_64 2:18.2.1-0.el9 ceph 171 k 2024-04-06T02:40:45.354 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:45.354 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-04-06T02:40:45.354 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-04-06T02:40:45.354 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-04-06T02:40:45.354 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:45.354 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 171 k 2024-04-06T02:40:45.355 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 494 k 2024-04-06T02:40:45.355 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-04-06T02:40:45.405 INFO:teuthology.orchestra.run.smithi184.stdout:rbd-nbd-18.2.1-0.el9.x86_64.rpm 3.3 MB/s | 171 kB 00:00 2024-04-06T02:40:45.406 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:40:45.406 INFO:teuthology.orchestra.run.smithi184.stdout:Total 3.2 MB/s | 171 kB 00:00 2024-04-06T02:40:45.406 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-04-06T02:40:45.414 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-04-06T02:40:45.414 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-04-06T02:40:45.473 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-04-06T02:40:45.473 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-04-06T02:40:45.626 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-04-06T02:40:45.722 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:46.149 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:46.463 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-04-06T02:40:46.463 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:46.463 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-04-06T02:40:46.463 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-nbd-2:18.2.1-0.el9.x86_64 2024-04-06T02:40:46.463 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:40:46.463 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-04-06T02:40:46.691 DEBUG:teuthology.parallel:result is None 2024-04-06T02:40:46.692 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-06T02:40:46.692 INFO:teuthology.packaging:ref: None 2024-04-06T02:40:46.692 INFO:teuthology.packaging:tag: v18.2.1 2024-04-06T02:40:46.692 INFO:teuthology.packaging:branch: None 2024-04-06T02:40:46.692 INFO:teuthology.packaging:sha1: 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T02:40:46.693 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-04-06T02:40:46.844 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-06T02:40:46.870 INFO:teuthology.orchestra.run.smithi105.stdout:18.2.1-0.el9 2024-04-06T02:40:46.871 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-0.el9 2024-04-06T02:40:46.871 INFO:teuthology.task.install:The correct ceph version 18.2.1-0 is installed. 2024-04-06T02:40:46.873 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-06T02:40:46.873 INFO:teuthology.packaging:ref: None 2024-04-06T02:40:46.873 INFO:teuthology.packaging:tag: v18.2.1 2024-04-06T02:40:46.873 INFO:teuthology.packaging:branch: None 2024-04-06T02:40:46.873 INFO:teuthology.packaging:sha1: 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T02:40:46.873 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-04-06T02:40:47.091 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-06T02:40:47.112 INFO:teuthology.orchestra.run.smithi184.stdout:18.2.1-0.el9 2024-04-06T02:40:47.112 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-0.el9 2024-04-06T02:40:47.112 INFO:teuthology.task.install:The correct ceph version 18.2.1-0 is installed. 2024-04-06T02:40:47.114 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-06T02:40:47.114 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-06T02:40:47.114 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-06T02:40:47.147 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-04-06T02:40:47.147 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-06T02:40:47.179 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-06T02:40:47.180 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-06T02:40:47.180 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/daemon-helper 2024-04-06T02:40:47.217 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-06T02:40:47.287 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-04-06T02:40:47.287 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/daemon-helper 2024-04-06T02:40:47.314 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-06T02:40:47.381 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-06T02:40:47.381 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-06T02:40:47.381 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-06T02:40:47.411 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-06T02:40:47.482 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-04-06T02:40:47.482 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-06T02:40:47.509 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-06T02:40:47.575 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-06T02:40:47.576 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-06T02:40:47.576 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/stdin-killer 2024-04-06T02:40:47.606 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-06T02:40:47.677 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-04-06T02:40:47.677 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/stdin-killer 2024-04-06T02:40:47.704 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-06T02:40:47.771 INFO:teuthology.run_tasks:Running task print... 2024-04-06T02:40:47.783 INFO:teuthology.task.print:**** done install task... 2024-04-06T02:40:47.783 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-06T02:40:47.904 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.io/ceph/ceph:v18.2.1', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN'], 'sha1': '9f192350ba5fe6d0a365b251a8c2af61f56a5fa6'} 2024-04-06T02:40:47.904 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v18.2.1 2024-04-06T02:40:47.905 INFO:tasks.cephadm:Cluster fsid is 130df578-f3bf-11ee-b649-cb9ed24678a4 2024-04-06T02:40:47.905 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-06T02:40:47.905 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-04-06T02:40:47.905 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi105': '172.21.15.105', 'mon.smithi184': '172.21.15.184'} 2024-04-06T02:40:47.905 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-06T02:40:47.905 DEBUG:teuthology.orchestra.run.smithi105:> sudo hostname $(hostname -s) 2024-04-06T02:40:47.937 DEBUG:teuthology.orchestra.run.smithi184:> sudo hostname $(hostname -s) 2024-04-06T02:40:47.964 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-04-06T02:40:47.964 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T02:40:48.106 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '9f192350ba5fe6d0a365b251a8c2af61f56a5fa6', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54816/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2027-g9f192350', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2027.g9f192350'}, 'url': 'https://2.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-05 10:23:10.679938', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-04-06T02:40:48.506 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref reef, sha1 37f5b3f83d34cb4a4842e4c8101a9b19ec6b2830 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-04-06T02:40:48.507 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/reef/37f5b3f83d34cb4a4842e4c8101a9b19ec6b2830/centos/9/x86_64/flavors/default/cephadm 2024-04-06T02:40:48.507 DEBUG:teuthology.orchestra.run.smithi105:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/reef/37f5b3f83d34cb4a4842e4c8101a9b19ec6b2830/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-06T02:40:49.104 INFO:teuthology.orchestra.run.smithi105.stdout:-rw-r--r--. 1 ubuntu ubuntu 215334 Apr 6 02:40 /home/ubuntu/cephtest/cephadm 2024-04-06T02:40:49.104 DEBUG:teuthology.orchestra.run.smithi184:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/reef/37f5b3f83d34cb4a4842e4c8101a9b19ec6b2830/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-06T02:40:49.675 INFO:teuthology.orchestra.run.smithi184.stdout:-rw-r--r--. 1 ubuntu ubuntu 215334 Apr 6 02:40 /home/ubuntu/cephtest/cephadm 2024-04-06T02:40:49.675 DEBUG:teuthology.orchestra.run.smithi105:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-06T02:40:49.693 DEBUG:teuthology.orchestra.run.smithi184:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-06T02:40:49.722 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v18.2.1 on all hosts... 2024-04-06T02:40:49.722 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull 2024-04-06T02:40:49.735 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull 2024-04-06T02:40:50.541 INFO:teuthology.orchestra.run.smithi105.stderr:Pulling container image quay.io/ceph/ceph:v18.2.1... 2024-04-06T02:40:50.616 INFO:teuthology.orchestra.run.smithi184.stderr:Pulling container image quay.io/ceph/ceph:v18.2.1... 2024-04-06T02:41:07.467 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-04-06T02:41:07.467 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-04-06T02:41:07.467 INFO:teuthology.orchestra.run.smithi184.stdout: "image_id": "5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf", 2024-04-06T02:41:07.467 INFO:teuthology.orchestra.run.smithi184.stdout: "repo_digests": [ 2024-04-06T02:41:07.467 INFO:teuthology.orchestra.run.smithi184.stdout: "quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3", 2024-04-06T02:41:07.467 INFO:teuthology.orchestra.run.smithi184.stdout: "quay.io/ceph/ceph@sha256:e8e55db8b4fd270dbec25bc764437a2a3abb707971c4dba5f559fb83018049dc" 2024-04-06T02:41:07.467 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-04-06T02:41:07.468 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-04-06T02:41:23.058 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-06T02:41:23.058 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-04-06T02:41:23.058 INFO:teuthology.orchestra.run.smithi105.stdout: "image_id": "5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf", 2024-04-06T02:41:23.059 INFO:teuthology.orchestra.run.smithi105.stdout: "repo_digests": [ 2024-04-06T02:41:23.059 INFO:teuthology.orchestra.run.smithi105.stdout: "quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3", 2024-04-06T02:41:23.059 INFO:teuthology.orchestra.run.smithi105.stdout: "quay.io/ceph/ceph@sha256:e8e55db8b4fd270dbec25bc764437a2a3abb707971c4dba5f559fb83018049dc" 2024-04-06T02:41:23.059 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-06T02:41:23.059 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-06T02:41:23.076 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /etc/ceph 2024-04-06T02:41:23.127 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /etc/ceph 2024-04-06T02:41:23.161 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 777 /etc/ceph 2024-04-06T02:41:23.187 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 777 /etc/ceph 2024-04-06T02:41:23.231 INFO:tasks.cephadm:Writing seed config... 2024-04-06T02:41:23.232 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-04-06T02:41:23.232 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-04-06T02:41:23.232 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-06T02:41:23.232 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-06T02:41:23.232 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-06T02:41:23.232 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-06T02:41:23.232 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-06T02:41:23.232 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-06T02:41:23.232 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-06T02:41:23.232 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-06T02:41:23.232 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-06T02:41:23.233 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-06T02:41:23.233 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-06T02:41:23.233 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-06T02:41:23.233 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-06T02:41:23.233 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-06T02:41:23.233 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-06T02:41:23.233 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-06T02:41:23.233 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-06T02:41:23.233 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-06T02:41:23.233 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-06T02:41:23.233 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-06T02:41:23.233 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-06T02:41:23.233 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-04-06T02:41:23.233 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-06T02:41:23.233 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-06T02:41:23.234 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-06T02:41:23.234 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-06T02:41:23.234 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-06T02:41:23.234 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-06T02:41:23.234 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-06T02:41:23.234 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-06T02:41:23.234 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-06T02:41:23.234 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-06T02:41:23.234 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-06T02:41:23.234 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-06T02:41:23.234 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-06T02:41:23.234 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-06T02:41:23.234 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-06T02:41:23.234 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-06T02:41:23.234 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-06T02:41:23.235 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-06T02:41:23.235 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-06T02:41:23.235 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-06T02:41:23.250 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 130df578-f3bf-11ee-b649-cb9ed24678a4 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-06T02:41:23.250 DEBUG:teuthology.orchestra.run.smithi105:mon.smithi105> sudo journalctl -f -n 0 -u ceph-130df578-f3bf-11ee-b649-cb9ed24678a4@mon.smithi105.service 2024-04-06T02:41:23.293 INFO:tasks.cephadm:Bootstrapping... 2024-04-06T02:41:23.293 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 -v bootstrap --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.105 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-06T02:41:23.447 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-06T02:41:23.448 INFO:teuthology.orchestra.run.smithi105.stdout:cephadm ['--image', 'quay.io/ceph/ceph:v18.2.1', '-v', 'bootstrap', '--fsid', '130df578-f3bf-11ee-b649-cb9ed24678a4', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.105', '--skip-admin-label'] 2024-04-06T02:41:23.472 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-06T02:41:23.472 INFO:teuthology.orchestra.run.smithi105.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-06T02:41:23.472 INFO:teuthology.orchestra.run.smithi105.stdout:Verifying podman|docker is present... 2024-04-06T02:41:23.495 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-06T02:41:23.495 INFO:teuthology.orchestra.run.smithi105.stdout:Verifying lvm2 is present... 2024-04-06T02:41:23.495 INFO:teuthology.orchestra.run.smithi105.stdout:Verifying time synchronization is in place... 2024-04-06T02:41:23.502 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-06T02:41:23.502 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-06T02:41:23.508 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-06T02:41:23.508 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout inactive 2024-04-06T02:41:23.514 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout enabled 2024-04-06T02:41:23.521 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout active 2024-04-06T02:41:23.521 INFO:teuthology.orchestra.run.smithi105.stdout:Unit chronyd.service is enabled and running 2024-04-06T02:41:23.521 INFO:teuthology.orchestra.run.smithi105.stdout:Repeating the final host check... 2024-04-06T02:41:23.543 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-06T02:41:23.543 INFO:teuthology.orchestra.run.smithi105.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-06T02:41:23.543 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl is present 2024-04-06T02:41:23.543 INFO:teuthology.orchestra.run.smithi105.stdout:lvcreate is present 2024-04-06T02:41:23.549 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-06T02:41:23.549 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-06T02:41:23.555 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-06T02:41:23.555 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout inactive 2024-04-06T02:41:23.561 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout enabled 2024-04-06T02:41:23.567 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout active 2024-04-06T02:41:23.567 INFO:teuthology.orchestra.run.smithi105.stdout:Unit chronyd.service is enabled and running 2024-04-06T02:41:23.567 INFO:teuthology.orchestra.run.smithi105.stdout:Host looks OK 2024-04-06T02:41:23.567 INFO:teuthology.orchestra.run.smithi105.stdout:Cluster fsid: 130df578-f3bf-11ee-b649-cb9ed24678a4 2024-04-06T02:41:23.567 INFO:teuthology.orchestra.run.smithi105.stdout:Acquiring lock 139790047166816 on /run/cephadm/130df578-f3bf-11ee-b649-cb9ed24678a4.lock 2024-04-06T02:41:23.567 INFO:teuthology.orchestra.run.smithi105.stdout:Lock 139790047166816 acquired on /run/cephadm/130df578-f3bf-11ee-b649-cb9ed24678a4.lock 2024-04-06T02:41:23.568 INFO:teuthology.orchestra.run.smithi105.stdout:Verifying IP 172.21.15.105 port 3300 ... 2024-04-06T02:41:23.568 INFO:teuthology.orchestra.run.smithi105.stdout:Verifying IP 172.21.15.105 port 6789 ... 2024-04-06T02:41:23.568 INFO:teuthology.orchestra.run.smithi105.stdout:Base mon IP(s) is [172.21.15.105:3300, 172.21.15.105:6789], mon addrv is [v2:172.21.15.105:3300,v1:172.21.15.105:6789] 2024-04-06T02:41:23.571 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.105 metric 100 2024-04-06T02:41:23.571 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.105 metric 100 2024-04-06T02:41:23.573 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-06T02:41:23.573 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-04-06T02:41:23.574 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1765sec hoplimit 64 pref medium 2024-04-06T02:41:23.576 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-06T02:41:23.576 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-06T02:41:23.576 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-06T02:41:23.576 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-06T02:41:23.576 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d429/64 scope link 2024-04-06T02:41:23.576 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-06T02:41:23.577 INFO:teuthology.orchestra.run.smithi105.stdout:Mon IP `172.21.15.105` is in CIDR network `172.21.0.0/20` 2024-04-06T02:41:23.577 INFO:teuthology.orchestra.run.smithi105.stdout:Mon IP `172.21.15.105` is in CIDR network `172.21.0.0/20` 2024-04-06T02:41:23.577 INFO:teuthology.orchestra.run.smithi105.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-06T02:41:23.577 INFO:teuthology.orchestra.run.smithi105.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-06T02:41:23.578 INFO:teuthology.orchestra.run.smithi105.stdout:Pulling container image quay.io/ceph/ceph:v18.2.1... 2024-04-06T02:41:24.378 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stdout 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf 2024-04-06T02:41:24.378 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stderr Trying to pull quay.io/ceph/ceph:v18.2.1... 2024-04-06T02:41:24.378 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stderr Getting image source signatures 2024-04-06T02:41:24.378 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stderr Copying blob sha256:a733d3c618b71f19c168ebecd1953429dce2c1631835ca182e9551c36dce5989 2024-04-06T02:41:24.378 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stderr Copying blob sha256:7feca07754707458c3945cf0062cf4dabc512f6d90fe1a9a1370b362b6011124 2024-04-06T02:41:24.378 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stderr Copying config sha256:5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf 2024-04-06T02:41:24.378 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-06T02:41:25.358 INFO:teuthology.orchestra.run.smithi105.stdout:ceph: stdout ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) 2024-04-06T02:41:25.358 INFO:teuthology.orchestra.run.smithi105.stdout:Ceph version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) 2024-04-06T02:41:25.358 INFO:teuthology.orchestra.run.smithi105.stdout:Extracting ceph user uid/gid from container image... 2024-04-06T02:41:26.363 INFO:teuthology.orchestra.run.smithi105.stdout:stat: stdout 167 167 2024-04-06T02:41:26.363 INFO:teuthology.orchestra.run.smithi105.stdout:Creating initial keys... 2024-04-06T02:41:27.345 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph-authtool: stdout AQBWthBm+lsQKxAAl76MRcc5N8ZMwrEAjZNP7w== 2024-04-06T02:41:28.341 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph-authtool: stdout AQBXthBmNMf5JxAADPpCw4KAlu6M5NiSqJj63Q== 2024-04-06T02:41:29.368 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph-authtool: stdout AQBYthBmTr3MJxAAmfb0ZpcTIteBAtIgO6yGGA== 2024-04-06T02:41:29.369 INFO:teuthology.orchestra.run.smithi105.stdout:Creating initial monmap... 2024-04-06T02:41:30.298 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-06T02:41:30.298 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-04-06T02:41:30.298 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 130df578-f3bf-11ee-b649-cb9ed24678a4 2024-04-06T02:41:30.298 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-06T02:41:30.298 INFO:teuthology.orchestra.run.smithi105.stdout:monmaptool for smithi105 [v2:172.21.15.105:3300,v1:172.21.15.105:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-06T02:41:30.298 INFO:teuthology.orchestra.run.smithi105.stdout:setting min_mon_release = pacific 2024-04-06T02:41:30.298 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/monmaptool: set fsid to 130df578-f3bf-11ee-b649-cb9ed24678a4 2024-04-06T02:41:30.298 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-06T02:41:30.298 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:41:30.298 INFO:teuthology.orchestra.run.smithi105.stdout:Creating mon... 2024-04-06T02:41:31.221 INFO:teuthology.orchestra.run.smithi105.stdout:create mon.smithi105 on 2024-04-06T02:41:31.432 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-06T02:41:31.574 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-06T02:41:31.745 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-130df578-f3bf-11ee-b649-cb9ed24678a4.target → /etc/systemd/system/ceph-130df578-f3bf-11ee-b649-cb9ed24678a4.target. 2024-04-06T02:41:31.745 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-130df578-f3bf-11ee-b649-cb9ed24678a4.target → /etc/systemd/system/ceph-130df578-f3bf-11ee-b649-cb9ed24678a4.target. 2024-04-06T02:41:31.968 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-130df578-f3bf-11ee-b649-cb9ed24678a4@mon.smithi105 2024-04-06T02:41:31.968 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Failed to reset failed state of unit ceph-130df578-f3bf-11ee-b649-cb9ed24678a4@mon.smithi105.service: Unit ceph-130df578-f3bf-11ee-b649-cb9ed24678a4@mon.smithi105.service not loaded. 2024-04-06T02:41:32.130 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-130df578-f3bf-11ee-b649-cb9ed24678a4.target.wants/ceph-130df578-f3bf-11ee-b649-cb9ed24678a4@mon.smithi105.service → /etc/systemd/system/ceph-130df578-f3bf-11ee-b649-cb9ed24678a4@.service. 2024-04-06T02:41:32.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:32 smithi105 podman[28841]: 2024-04-06 02:41:32.28400423 +0000 UTC m=+0.017896056 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph:v18.2.1 2024-04-06T02:41:32.651 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-06T02:41:32.651 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout disabled 2024-04-06T02:41:32.658 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-06T02:41:32.658 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout inactive 2024-04-06T02:41:32.658 INFO:teuthology.orchestra.run.smithi105.stdout:firewalld.service is not enabled 2024-04-06T02:41:32.658 INFO:teuthology.orchestra.run.smithi105.stdout:Not possible to enable service . firewalld.service is not available 2024-04-06T02:41:32.659 INFO:teuthology.orchestra.run.smithi105.stdout:Waiting for mon to start... 2024-04-06T02:41:32.659 INFO:teuthology.orchestra.run.smithi105.stdout:Waiting for mon... 2024-04-06T02:41:32.748 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:32 smithi105 podman[28841]: 2024-04-06 02:41:32.580573011 +0000 UTC m=+0.314464836 container init 727e5d63796a4e053b286557daea29844a45d16d2c92a48f54d1389a9d59c9f2 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105, RELEASE=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True) 2024-04-06T02:41:32.748 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:32 smithi105 podman[28841]: 2024-04-06 02:41:32.583375629 +0000 UTC m=+0.317267457 container start 727e5d63796a4e053b286557daea29844a45d16d2c92a48f54d1389a9d59c9f2 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240222, RELEASE=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-06T02:41:32.748 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:32 smithi105 ceph-mon[28859]: mkfs 130df578-f3bf-11ee-b649-cb9ed24678a4 2024-04-06T02:41:32.749 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:32 smithi105 ceph-mon[28859]: mon.smithi105 is new leader, mons smithi105 in quorum (ranks 0) 2024-04-06T02:41:32.749 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:32 smithi105 bash[28841]: 727e5d63796a4e053b286557daea29844a45d16d2c92a48f54d1389a9d59c9f2 2024-04-06T02:41:32.749 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:32 smithi105 systemd[1]: Started Ceph mon.smithi105 for 130df578-f3bf-11ee-b649-cb9ed24678a4. 2024-04-06T02:41:33.670 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout cluster: 2024-04-06T02:41:33.671 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout id: 130df578-f3bf-11ee-b649-cb9ed24678a4 2024-04-06T02:41:33.671 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-06T02:41:33.671 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-04-06T02:41:33.671 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout services: 2024-04-06T02:41:33.671 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi105 (age 0.503713s) 2024-04-06T02:41:33.671 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-06T02:41:33.671 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-06T02:41:33.671 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-04-06T02:41:33.671 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout data: 2024-04-06T02:41:33.671 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-06T02:41:33.671 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-06T02:41:33.671 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-06T02:41:33.671 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout pgs: 2024-04-06T02:41:33.671 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-04-06T02:41:33.672 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.071+0000 7f87e3db8700 1 Processor -- start 2024-04-06T02:41:33.672 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.071+0000 7f87e3db8700 1 -- start start 2024-04-06T02:41:33.672 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.071+0000 7f87e3db8700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f87dc104e70 0x7f87dc105290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:41:33.672 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.071+0000 7f87e3db8700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc079df0 con 0x7f87dc104e70 2024-04-06T02:41:33.672 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.072+0000 7f87e1b54700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f87dc104e70 0x7f87dc105290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:41:33.672 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.072+0000 7f87e1b54700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f87dc104e70 0x7f87dc105290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33850/0 (socket says 172.21.15.105:33850) 2024-04-06T02:41:33.672 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.072+0000 7f87e1b54700 1 -- 172.21.15.105:0/651685700 learned_addr learned my addr 172.21.15.105:0/651685700 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:41:33.672 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.072+0000 7f87e1b54700 1 -- 172.21.15.105:0/651685700 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87dc079f30 con 0x7f87dc104e70 2024-04-06T02:41:33.672 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.072+0000 7f87e1b54700 1 --2- 172.21.15.105:0/651685700 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f87dc104e70 0x7f87dc105290 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f87d4009e80 tx=0x7f87d400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcabda2efa56f3e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:41:33.672 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.072+0000 7f87e0b52700 1 -- 172.21.15.105:0/651685700 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87d400e070 con 0x7f87dc104e70 2024-04-06T02:41:33.672 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.072+0000 7f87e0b52700 1 -- 172.21.15.105:0/651685700 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d4003a20 con 0x7f87dc104e70 2024-04-06T02:41:33.672 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.072+0000 7f87e0b52700 1 -- 172.21.15.105:0/651685700 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87d4003bd0 con 0x7f87dc104e70 2024-04-06T02:41:33.672 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.073+0000 7f87e3db8700 1 -- 172.21.15.105:0/651685700 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f87dc104e70 msgr2=0x7f87dc105290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:41:33.673 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.073+0000 7f87e3db8700 1 --2- 172.21.15.105:0/651685700 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f87dc104e70 0x7f87dc105290 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f87d4009e80 tx=0x7f87d400b330 comp rx=0 tx=0).stop 2024-04-06T02:41:33.673 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.073+0000 7f87e3db8700 1 -- 172.21.15.105:0/651685700 shutdown_connections 2024-04-06T02:41:33.673 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.073+0000 7f87e3db8700 1 --2- 172.21.15.105:0/651685700 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f87dc104e70 0x7f87dc105290 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:41:33.673 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.073+0000 7f87e3db8700 1 -- 172.21.15.105:0/651685700 >> 172.21.15.105:0/651685700 conn(0x7f87dc1009f0 msgr2=0x7f87dc102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:41:33.673 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.073+0000 7f87e3db8700 1 -- 172.21.15.105:0/651685700 shutdown_connections 2024-04-06T02:41:33.673 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.073+0000 7f87e3db8700 1 -- 172.21.15.105:0/651685700 wait complete. 2024-04-06T02:41:33.673 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.074+0000 7f87e3db8700 1 Processor -- start 2024-04-06T02:41:33.673 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.074+0000 7f87e3db8700 1 -- start start 2024-04-06T02:41:33.673 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.074+0000 7f87e3db8700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f87dc104e70 0x7f87dc19bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:41:33.673 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.074+0000 7f87e3db8700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc19c120 con 0x7f87dc104e70 2024-04-06T02:41:33.673 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.074+0000 7f87e1b54700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f87dc104e70 0x7f87dc19bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:41:33.673 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.074+0000 7f87e1b54700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f87dc104e70 0x7f87dc19bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33864/0 (socket says 172.21.15.105:33864) 2024-04-06T02:41:33.673 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.074+0000 7f87e1b54700 1 -- 172.21.15.105:0/3784601803 learned_addr learned my addr 172.21.15.105:0/3784601803 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:41:33.673 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.074+0000 7f87e1b54700 1 -- 172.21.15.105:0/3784601803 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d4009b30 con 0x7f87dc104e70 2024-04-06T02:41:33.674 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.074+0000 7f87e1b54700 1 --2- 172.21.15.105:0/3784601803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f87dc104e70 0x7f87dc19bbe0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f87dc079230 tx=0x7f87d4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:41:33.674 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.074+0000 7f87d2ffd700 1 -- 172.21.15.105:0/3784601803 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87d4004240 con 0x7f87dc104e70 2024-04-06T02:41:33.674 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.075+0000 7f87d2ffd700 1 -- 172.21.15.105:0/3784601803 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d40043e0 con 0x7f87dc104e70 2024-04-06T02:41:33.674 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.075+0000 7f87e3db8700 1 -- 172.21.15.105:0/3784601803 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87dc19c320 con 0x7f87dc104e70 2024-04-06T02:41:33.674 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.075+0000 7f87d2ffd700 1 -- 172.21.15.105:0/3784601803 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87d40046e0 con 0x7f87dc104e70 2024-04-06T02:41:33.674 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.075+0000 7f87e3db8700 1 -- 172.21.15.105:0/3784601803 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87dc19c7c0 con 0x7f87dc104e70 2024-04-06T02:41:33.674 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.075+0000 7f87d2ffd700 1 -- 172.21.15.105:0/3784601803 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f87d4019050 con 0x7f87dc104e70 2024-04-06T02:41:33.674 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.075+0000 7f87d2ffd700 1 -- 172.21.15.105:0/3784601803 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f87d40257e0 con 0x7f87dc104e70 2024-04-06T02:41:33.674 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.075+0000 7f87e3db8700 1 -- 172.21.15.105:0/3784601803 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87dc195900 con 0x7f87dc104e70 2024-04-06T02:41:33.674 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.078+0000 7f87d2ffd700 1 -- 172.21.15.105:0/3784601803 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f87d402a060 con 0x7f87dc104e70 2024-04-06T02:41:33.674 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.146+0000 7f87e3db8700 1 -- 172.21.15.105:0/3784601803 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f87dc02cee0 con 0x7f87dc104e70 2024-04-06T02:41:33.674 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.147+0000 7f87d2ffd700 1 -- 172.21.15.105:0/3784601803 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f87d4020030 con 0x7f87dc104e70 2024-04-06T02:41:33.674 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.147+0000 7f87e3db8700 1 -- 172.21.15.105:0/3784601803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f87dc104e70 msgr2=0x7f87dc19bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:41:33.675 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.147+0000 7f87e3db8700 1 --2- 172.21.15.105:0/3784601803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f87dc104e70 0x7f87dc19bbe0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f87dc079230 tx=0x7f87d4004ad0 comp rx=0 tx=0).stop 2024-04-06T02:41:33.675 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.147+0000 7f87e3db8700 1 -- 172.21.15.105:0/3784601803 shutdown_connections 2024-04-06T02:41:33.675 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.147+0000 7f87e3db8700 1 --2- 172.21.15.105:0/3784601803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f87dc104e70 0x7f87dc19bbe0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:41:33.675 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.147+0000 7f87e3db8700 1 -- 172.21.15.105:0/3784601803 >> 172.21.15.105:0/3784601803 conn(0x7f87dc1009f0 msgr2=0x7f87dc192a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:41:33.675 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.147+0000 7f87e3db8700 1 -- 172.21.15.105:0/3784601803 shutdown_connections 2024-04-06T02:41:33.675 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:33.147+0000 7f87e3db8700 1 -- 172.21.15.105:0/3784601803 wait complete. 2024-04-06T02:41:33.675 INFO:teuthology.orchestra.run.smithi105.stdout:mon is available 2024-04-06T02:41:33.675 INFO:teuthology.orchestra.run.smithi105.stdout:Assimilating anything we can from ceph.conf... 2024-04-06T02:41:33.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:33 smithi105 ceph-mon[28859]: mon.smithi105 is new leader, mons smithi105 in quorum (ranks 0) 2024-04-06T02:41:33.676 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:33 smithi105 ceph-mon[28859]: monmap e1: 1 mons at {smithi105=[v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-06T02:41:33.676 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:33 smithi105 ceph-mon[28859]: fsmap 2024-04-06T02:41:33.676 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:33 smithi105 ceph-mon[28859]: osdmap e1: 0 total, 0 up, 0 in 2024-04-06T02:41:33.676 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:33 smithi105 ceph-mon[28859]: mgrmap e1: no daemons active 2024-04-06T02:41:33.676 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:33 smithi105 ceph-mon[28859]: from='client.? 172.21.15.105:0/3784601803' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-06T02:41:34.636 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-04-06T02:41:34.636 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout [global] 2024-04-06T02:41:34.636 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout fsid = 130df578-f3bf-11ee-b649-cb9ed24678a4 2024-04-06T02:41:34.636 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.105:3300,v1:172.21.15.105:6789] 2024-04-06T02:41:34.636 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-06T02:41:34.636 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-06T02:41:34.636 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-06T02:41:34.636 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-06T02:41:34.636 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-04-06T02:41:34.636 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-06T02:41:34.637 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-06T02:41:34.637 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-04-06T02:41:34.637 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout [osd] 2024-04-06T02:41:34.637 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-06T02:41:34.637 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-06T02:41:34.637 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-06T02:41:34.637 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-06T02:41:34.637 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-06T02:41:34.637 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.058+0000 7fc7cf1ac700 1 Processor -- start 2024-04-06T02:41:34.637 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.058+0000 7fc7cf1ac700 1 -- start start 2024-04-06T02:41:34.637 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.058+0000 7fc7cf1ac700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7c8105450 0x7fc7c8105870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:41:34.637 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.058+0000 7fc7cf1ac700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c8105e40 con 0x7fc7c8105450 2024-04-06T02:41:34.637 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.059+0000 7fc7ccf48700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7c8105450 0x7fc7c8105870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:41:34.638 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.059+0000 7fc7ccf48700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7c8105450 0x7fc7c8105870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33880/0 (socket says 172.21.15.105:33880) 2024-04-06T02:41:34.638 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.059+0000 7fc7ccf48700 1 -- 172.21.15.105:0/1047404502 learned_addr learned my addr 172.21.15.105:0/1047404502 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:41:34.638 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.059+0000 7fc7ccf48700 1 -- 172.21.15.105:0/1047404502 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c8106650 con 0x7fc7c8105450 2024-04-06T02:41:34.638 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.059+0000 7fc7ccf48700 1 --2- 172.21.15.105:0/1047404502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7c8105450 0x7fc7c8105870 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4009e80 tx=0x7fc7c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c1157f83d08583f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:41:34.638 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.059+0000 7fc7bf7fe700 1 -- 172.21.15.105:0/1047404502 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7c400e070 con 0x7fc7c8105450 2024-04-06T02:41:34.638 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.059+0000 7fc7bf7fe700 1 -- 172.21.15.105:0/1047404502 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c4003a20 con 0x7fc7c8105450 2024-04-06T02:41:34.638 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.059+0000 7fc7bf7fe700 1 -- 172.21.15.105:0/1047404502 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7c4003bd0 con 0x7fc7c8105450 2024-04-06T02:41:34.638 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.060+0000 7fc7cf1ac700 1 -- 172.21.15.105:0/1047404502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7c8105450 msgr2=0x7fc7c8105870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:41:34.638 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.060+0000 7fc7cf1ac700 1 --2- 172.21.15.105:0/1047404502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7c8105450 0x7fc7c8105870 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4009e80 tx=0x7fc7c400b330 comp rx=0 tx=0).stop 2024-04-06T02:41:34.638 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.060+0000 7fc7cf1ac700 1 -- 172.21.15.105:0/1047404502 shutdown_connections 2024-04-06T02:41:34.638 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.060+0000 7fc7cf1ac700 1 --2- 172.21.15.105:0/1047404502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7c8105450 0x7fc7c8105870 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:41:34.638 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.060+0000 7fc7cf1ac700 1 -- 172.21.15.105:0/1047404502 >> 172.21.15.105:0/1047404502 conn(0x7fc7c81009b0 msgr2=0x7fc7c8102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:41:34.639 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.060+0000 7fc7cf1ac700 1 -- 172.21.15.105:0/1047404502 shutdown_connections 2024-04-06T02:41:34.639 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.060+0000 7fc7cf1ac700 1 -- 172.21.15.105:0/1047404502 wait complete. 2024-04-06T02:41:34.639 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.060+0000 7fc7cf1ac700 1 Processor -- start 2024-04-06T02:41:34.639 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.060+0000 7fc7cf1ac700 1 -- start start 2024-04-06T02:41:34.639 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.060+0000 7fc7cf1ac700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7c8105450 0x7fc7c8199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:41:34.639 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.060+0000 7fc7cf1ac700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c8199f50 con 0x7fc7c8105450 2024-04-06T02:41:34.639 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.061+0000 7fc7ccf48700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7c8105450 0x7fc7c8199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:41:34.639 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.061+0000 7fc7ccf48700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7c8105450 0x7fc7c8199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33896/0 (socket says 172.21.15.105:33896) 2024-04-06T02:41:34.639 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.061+0000 7fc7ccf48700 1 -- 172.21.15.105:0/910987723 learned_addr learned my addr 172.21.15.105:0/910987723 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:41:34.639 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.061+0000 7fc7ccf48700 1 -- 172.21.15.105:0/910987723 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c4009b30 con 0x7fc7c8105450 2024-04-06T02:41:34.639 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.061+0000 7fc7ccf48700 1 --2- 172.21.15.105:0/910987723 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7c8105450 0x7fc7c8199a10 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4004000 tx=0x7fc7c4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:41:34.639 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.061+0000 7fc7bdffb700 1 -- 172.21.15.105:0/910987723 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7c4004240 con 0x7fc7c8105450 2024-04-06T02:41:34.639 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.061+0000 7fc7bdffb700 1 -- 172.21.15.105:0/910987723 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c40043e0 con 0x7fc7c8105450 2024-04-06T02:41:34.639 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.061+0000 7fc7cf1ac700 1 -- 172.21.15.105:0/910987723 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7c819a150 con 0x7fc7c8105450 2024-04-06T02:41:34.639 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.061+0000 7fc7bdffb700 1 -- 172.21.15.105:0/910987723 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7c40046e0 con 0x7fc7c8105450 2024-04-06T02:41:34.640 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.061+0000 7fc7cf1ac700 1 -- 172.21.15.105:0/910987723 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7c819a5f0 con 0x7fc7c8105450 2024-04-06T02:41:34.640 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.061+0000 7fc7bdffb700 1 -- 172.21.15.105:0/910987723 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc7c4019050 con 0x7fc7c8105450 2024-04-06T02:41:34.640 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.061+0000 7fc7bdffb700 1 -- 172.21.15.105:0/910987723 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc7c4025790 con 0x7fc7c8105450 2024-04-06T02:41:34.640 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.062+0000 7fc7cf1ac700 1 -- 172.21.15.105:0/910987723 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7c81932f0 con 0x7fc7c8105450 2024-04-06T02:41:34.640 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.064+0000 7fc7bdffb700 1 -- 172.21.15.105:0/910987723 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fc7c4044b10 con 0x7fc7c8105450 2024-04-06T02:41:34.640 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.132+0000 7fc7cf1ac700 1 -- 172.21.15.105:0/910987723 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fc7c80623c0 con 0x7fc7c8105450 2024-04-06T02:41:34.640 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.135+0000 7fc7bdffb700 1 -- 172.21.15.105:0/910987723 <== mon.0 v2:172.21.15.105:3300/0 7 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc7c4012d80 con 0x7fc7c8105450 2024-04-06T02:41:34.640 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.135+0000 7fc7bdffb700 1 -- 172.21.15.105:0/910987723 <== mon.0 v2:172.21.15.105:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7fc7c4033020 con 0x7fc7c8105450 2024-04-06T02:41:34.640 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.136+0000 7fc7cf1ac700 1 -- 172.21.15.105:0/910987723 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7c8105450 msgr2=0x7fc7c8199a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:41:34.640 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.136+0000 7fc7cf1ac700 1 --2- 172.21.15.105:0/910987723 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7c8105450 0x7fc7c8199a10 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4004000 tx=0x7fc7c4004ad0 comp rx=0 tx=0).stop 2024-04-06T02:41:34.640 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.136+0000 7fc7cf1ac700 1 -- 172.21.15.105:0/910987723 shutdown_connections 2024-04-06T02:41:34.640 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.136+0000 7fc7cf1ac700 1 --2- 172.21.15.105:0/910987723 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7c8105450 0x7fc7c8199a10 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:41:34.640 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.136+0000 7fc7cf1ac700 1 -- 172.21.15.105:0/910987723 >> 172.21.15.105:0/910987723 conn(0x7fc7c81009b0 msgr2=0x7fc7c8079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:41:34.641 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.136+0000 7fc7cf1ac700 1 -- 172.21.15.105:0/910987723 shutdown_connections 2024-04-06T02:41:34.641 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:34.136+0000 7fc7cf1ac700 1 -- 172.21.15.105:0/910987723 wait complete. 2024-04-06T02:41:34.641 INFO:teuthology.orchestra.run.smithi105.stdout:Generating new minimal ceph.conf... 2024-04-06T02:41:34.889 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:34 smithi105 ceph-mon[28859]: from='client.? 172.21.15.105:0/910987723' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-06T02:41:34.889 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:34 smithi105 ceph-mon[28859]: from='client.? 172.21.15.105:0/910987723' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-06T02:41:35.632 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.076+0000 7feef95ec700 1 Processor -- start 2024-04-06T02:41:35.632 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.076+0000 7feef95ec700 1 -- start start 2024-04-06T02:41:35.632 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.076+0000 7feef95ec700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feef4107020 0x7feef4109410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:41:35.632 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.076+0000 7feef95ec700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef4079e30 con 0x7feef4107020 2024-04-06T02:41:35.632 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.076+0000 7feef2ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feef4107020 0x7feef4109410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:41:35.632 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.076+0000 7feef2ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feef4107020 0x7feef4109410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33906/0 (socket says 172.21.15.105:33906) 2024-04-06T02:41:35.632 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.076+0000 7feef2ffd700 1 -- 172.21.15.105:0/4073167006 learned_addr learned my addr 172.21.15.105:0/4073167006 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:41:35.632 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.076+0000 7feef2ffd700 1 -- 172.21.15.105:0/4073167006 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef4109950 con 0x7feef4107020 2024-04-06T02:41:35.632 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.076+0000 7feef2ffd700 1 --2- 172.21.15.105:0/4073167006 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feef4107020 0x7feef4109410 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7feee40098a0 tx=0x7feee4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4c33df08924af37 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:41:35.632 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.077+0000 7feef1ffb700 1 -- 172.21.15.105:0/4073167006 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feee4004180 con 0x7feef4107020 2024-04-06T02:41:35.633 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.077+0000 7feef1ffb700 1 -- 172.21.15.105:0/4073167006 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7feee40042e0 con 0x7feef4107020 2024-04-06T02:41:35.633 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.077+0000 7feef1ffb700 1 -- 172.21.15.105:0/4073167006 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feee4011a50 con 0x7feef4107020 2024-04-06T02:41:35.633 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.077+0000 7feef95ec700 1 -- 172.21.15.105:0/4073167006 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feef4107020 msgr2=0x7feef4109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:41:35.633 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.077+0000 7feef95ec700 1 --2- 172.21.15.105:0/4073167006 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feef4107020 0x7feef4109410 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7feee40098a0 tx=0x7feee4009bb0 comp rx=0 tx=0).stop 2024-04-06T02:41:35.633 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.077+0000 7feef95ec700 1 -- 172.21.15.105:0/4073167006 shutdown_connections 2024-04-06T02:41:35.633 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.077+0000 7feef95ec700 1 --2- 172.21.15.105:0/4073167006 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feef4107020 0x7feef4109410 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:41:35.633 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.077+0000 7feef95ec700 1 -- 172.21.15.105:0/4073167006 >> 172.21.15.105:0/4073167006 conn(0x7feef41009b0 msgr2=0x7feef4102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:41:35.633 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.077+0000 7feef95ec700 1 -- 172.21.15.105:0/4073167006 shutdown_connections 2024-04-06T02:41:35.633 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.077+0000 7feef95ec700 1 -- 172.21.15.105:0/4073167006 wait complete. 2024-04-06T02:41:35.633 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.078+0000 7feef95ec700 1 Processor -- start 2024-04-06T02:41:35.633 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.078+0000 7feef95ec700 1 -- start start 2024-04-06T02:41:35.633 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.078+0000 7feef95ec700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feef4107020 0x7feef419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:41:35.633 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.078+0000 7feef95ec700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef419a750 con 0x7feef4107020 2024-04-06T02:41:35.634 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.078+0000 7feef2ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feef4107020 0x7feef419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:41:35.634 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.078+0000 7feef2ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feef4107020 0x7feef419a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33916/0 (socket says 172.21.15.105:33916) 2024-04-06T02:41:35.634 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.078+0000 7feef2ffd700 1 -- 172.21.15.105:0/1264285917 learned_addr learned my addr 172.21.15.105:0/1264285917 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:41:35.634 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.079+0000 7feef2ffd700 1 -- 172.21.15.105:0/1264285917 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee4009550 con 0x7feef4107020 2024-04-06T02:41:35.634 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.079+0000 7feef2ffd700 1 --2- 172.21.15.105:0/1264285917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feef4107020 0x7feef419a210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7feee4003f80 tx=0x7feee4011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:41:35.634 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.079+0000 7feedbfff700 1 -- 172.21.15.105:0/1264285917 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feee4003a80 con 0x7feef4107020 2024-04-06T02:41:35.634 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.079+0000 7feedbfff700 1 -- 172.21.15.105:0/1264285917 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7feee4003be0 con 0x7feef4107020 2024-04-06T02:41:35.634 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.079+0000 7feef95ec700 1 -- 172.21.15.105:0/1264285917 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef419a950 con 0x7feef4107020 2024-04-06T02:41:35.634 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.079+0000 7feedbfff700 1 -- 172.21.15.105:0/1264285917 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feee4019820 con 0x7feef4107020 2024-04-06T02:41:35.634 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.079+0000 7feef95ec700 1 -- 172.21.15.105:0/1264285917 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef419adf0 con 0x7feef4107020 2024-04-06T02:41:35.634 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.079+0000 7feedbfff700 1 -- 172.21.15.105:0/1264285917 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7feee402d440 con 0x7feef4107020 2024-04-06T02:41:35.634 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.079+0000 7feedbfff700 1 -- 172.21.15.105:0/1264285917 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7feee40247c0 con 0x7feef4107020 2024-04-06T02:41:35.635 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.080+0000 7feef95ec700 1 -- 172.21.15.105:0/1264285917 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef40623c0 con 0x7feef4107020 2024-04-06T02:41:35.635 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.082+0000 7feedbfff700 1 -- 172.21.15.105:0/1264285917 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7feee401acc0 con 0x7feef4107020 2024-04-06T02:41:35.635 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.148+0000 7feef95ec700 1 -- 172.21.15.105:0/1264285917 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7feef419d980 con 0x7feef4107020 2024-04-06T02:41:35.635 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.149+0000 7feedbfff700 1 -- 172.21.15.105:0/1264285917 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7feee4035090 con 0x7feef4107020 2024-04-06T02:41:35.635 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.149+0000 7feef95ec700 1 -- 172.21.15.105:0/1264285917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feef4107020 msgr2=0x7feef419a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:41:35.635 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.149+0000 7feef95ec700 1 --2- 172.21.15.105:0/1264285917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feef4107020 0x7feef419a210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7feee4003f80 tx=0x7feee4011ab0 comp rx=0 tx=0).stop 2024-04-06T02:41:35.635 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.149+0000 7feef95ec700 1 -- 172.21.15.105:0/1264285917 shutdown_connections 2024-04-06T02:41:35.635 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.149+0000 7feef95ec700 1 --2- 172.21.15.105:0/1264285917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feef4107020 0x7feef419a210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:41:35.635 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.149+0000 7feef95ec700 1 -- 172.21.15.105:0/1264285917 >> 172.21.15.105:0/1264285917 conn(0x7feef41009b0 msgr2=0x7feef4102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:41:35.635 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.150+0000 7feef95ec700 1 -- 172.21.15.105:0/1264285917 shutdown_connections 2024-04-06T02:41:35.635 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:35.150+0000 7feef95ec700 1 -- 172.21.15.105:0/1264285917 wait complete. 2024-04-06T02:41:35.635 INFO:teuthology.orchestra.run.smithi105.stdout:Restarting the monitor... 2024-04-06T02:41:35.895 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:35 smithi105 ceph-mon[28859]: from='client.? 172.21.15.105:0/1264285917' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:41:35.895 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:35 smithi105 systemd[1]: Stopping Ceph mon.smithi105 for 130df578-f3bf-11ee-b649-cb9ed24678a4... 2024-04-06T02:41:36.173 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[28855]: 2024-04-06T02:41:35.911+0000 7fb5f4a16700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi105 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T02:41:36.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[28855]: 2024-04-06T02:41:35.911+0000 7fb5f4a16700 -1 mon.smithi105@0(leader) e1 *** Got Signal Terminated *** 2024-04-06T02:41:36.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:36 smithi105 podman[29102]: 2024-04-06 02:41:36.126369211 +0000 UTC m=+0.323501594 container died 727e5d63796a4e053b286557daea29844a45d16d2c92a48f54d1389a9d59c9f2 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=HEAD) 2024-04-06T02:41:36.673 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:36 smithi105 podman[29102]: 2024-04-06 02:41:36.46947527 +0000 UTC m=+0.666607653 container cleanup 727e5d63796a4e053b286557daea29844a45d16d2c92a48f54d1389a9d59c9f2 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-06T02:41:36.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:36 smithi105 bash[29102]: ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105 2024-04-06T02:41:37.063 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:36 smithi105 podman[29116]: 2024-04-06 02:41:36.718690034 +0000 UTC m=+0.595264367 container remove 727e5d63796a4e053b286557daea29844a45d16d2c92a48f54d1389a9d59c9f2 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS) 2024-04-06T02:41:37.063 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:36 smithi105 systemd[1]: ceph-130df578-f3bf-11ee-b649-cb9ed24678a4@mon.smithi105.service: Deactivated successfully. 2024-04-06T02:41:37.063 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:36 smithi105 systemd[1]: Stopped Ceph mon.smithi105 for 130df578-f3bf-11ee-b649-cb9ed24678a4. 2024-04-06T02:41:37.063 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:36 smithi105 systemd[1]: Starting Ceph mon.smithi105 for 130df578-f3bf-11ee-b649-cb9ed24678a4... 2024-04-06T02:41:37.316 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 podman[29209]: 2024-04-06 02:41:37.079824273 +0000 UTC m=+0.104683094 container create d27bf0498159c5eb042ec29dc651c2d3cde19f6c07dbc359b8c9870c0e665bda (image=quay.io/ceph/ceph:v18.2.1, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=HEAD) 2024-04-06T02:41:37.362 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 podman[29209]: 2024-04-06 02:41:36.993146115 +0000 UTC m=+0.018004945 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph:v18.2.1 2024-04-06T02:41:37.362 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 podman[29209]: 2024-04-06 02:41:37.248857393 +0000 UTC m=+0.273716213 container init d27bf0498159c5eb042ec29dc651c2d3cde19f6c07dbc359b8c9870c0e665bda (image=quay.io/ceph/ceph:v18.2.1, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , ceph=True, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222) 2024-04-06T02:41:37.362 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 podman[29209]: 2024-04-06 02:41:37.251910542 +0000 UTC m=+0.276769363 container start d27bf0498159c5eb042ec29dc651c2d3cde19f6c07dbc359b8c9870c0e665bda (image=quay.io/ceph/ceph:v18.2.1, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-06T02:41:37.362 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: set uid:gid to 167:167 (ceph:ceph) 2024-04-06T02:41:37.363 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable), process ceph-mon, pid 2 2024-04-06T02:41:37.363 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: pidfile_write: ignore empty --pid-file 2024-04-06T02:41:37.363 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: load: jerasure load: lrc 2024-04-06T02:41:37.363 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: RocksDB version: 7.9.2 2024-04-06T02:41:37.363 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Git sha 0 2024-04-06T02:41:37.363 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Compile date 2023-12-11 22:07:34 2024-04-06T02:41:37.363 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: DB SUMMARY 2024-04-06T02:41:37.363 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: DB Session ID: KYBD78FR1RLA65PXSTHK 2024-04-06T02:41:37.363 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: CURRENT file: CURRENT 2024-04-06T02:41:37.363 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: IDENTITY file: IDENTITY 2024-04-06T02:41:37.363 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-06T02:41:37.363 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi105/store.db dir, Total Num: 1, files: 000008.sst 2024-04-06T02:41:37.363 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi105/store.db: 000009.log size: 98383 ; 2024-04-06T02:41:37.363 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.error_if_exists: 0 2024-04-06T02:41:37.364 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.create_if_missing: 0 2024-04-06T02:41:37.364 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.paranoid_checks: 1 2024-04-06T02:41:37.365 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-06T02:41:37.365 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-06T02:41:37.366 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-06T02:41:37.366 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.env: 0x5566f6605720 2024-04-06T02:41:37.366 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.fs: PosixFileSystem 2024-04-06T02:41:37.366 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.info_log: 0x5566f85d52e0 2024-04-06T02:41:37.366 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.max_file_opening_threads: 16 2024-04-06T02:41:37.366 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.statistics: (nil) 2024-04-06T02:41:37.366 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.use_fsync: 0 2024-04-06T02:41:37.366 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.max_log_file_size: 0 2024-04-06T02:41:37.366 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-06T02:41:37.366 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-06T02:41:37.366 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.keep_log_file_num: 1000 2024-04-06T02:41:37.366 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.recycle_log_file_num: 0 2024-04-06T02:41:37.366 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.allow_fallocate: 1 2024-04-06T02:41:37.366 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.allow_mmap_reads: 0 2024-04-06T02:41:37.367 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.allow_mmap_writes: 0 2024-04-06T02:41:37.367 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.use_direct_reads: 0 2024-04-06T02:41:37.367 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-06T02:41:37.367 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.create_missing_column_families: 0 2024-04-06T02:41:37.367 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.db_log_dir: 2024-04-06T02:41:37.367 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.wal_dir: 2024-04-06T02:41:37.367 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-06T02:41:37.367 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-06T02:41:37.367 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-06T02:41:37.367 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-06T02:41:37.367 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-06T02:41:37.367 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-06T02:41:37.367 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.advise_random_on_open: 1 2024-04-06T02:41:37.368 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.db_write_buffer_size: 0 2024-04-06T02:41:37.368 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.write_buffer_manager: 0x5566f7846140 2024-04-06T02:41:37.368 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-06T02:41:37.368 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-06T02:41:37.368 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-06T02:41:37.368 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.rate_limiter: (nil) 2024-04-06T02:41:37.368 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-06T02:41:37.368 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.wal_recovery_mode: 2 2024-04-06T02:41:37.368 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.enable_thread_tracking: 0 2024-04-06T02:41:37.368 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.enable_pipelined_write: 0 2024-04-06T02:41:37.368 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.unordered_write: 0 2024-04-06T02:41:37.368 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-06T02:41:37.368 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-06T02:41:37.368 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-06T02:41:37.369 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-06T02:41:37.369 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.row_cache: None 2024-04-06T02:41:37.369 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.wal_filter: None 2024-04-06T02:41:37.369 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-06T02:41:37.369 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.allow_ingest_behind: 0 2024-04-06T02:41:37.369 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.two_write_queues: 0 2024-04-06T02:41:37.369 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.manual_wal_flush: 0 2024-04-06T02:41:37.369 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.wal_compression: 0 2024-04-06T02:41:37.369 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.atomic_flush: 0 2024-04-06T02:41:37.369 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-06T02:41:37.369 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-06T02:41:37.369 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-06T02:41:37.369 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.log_readahead_size: 0 2024-04-06T02:41:37.370 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-06T02:41:37.370 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.best_efforts_recovery: 0 2024-04-06T02:41:37.370 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-06T02:41:37.370 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-06T02:41:37.370 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.allow_data_in_errors: 0 2024-04-06T02:41:37.370 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.db_host_id: __hostname__ 2024-04-06T02:41:37.370 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-06T02:41:37.370 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.max_background_jobs: 2 2024-04-06T02:41:37.370 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.max_background_compactions: -1 2024-04-06T02:41:37.370 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.max_subcompactions: 1 2024-04-06T02:41:37.370 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-06T02:41:37.370 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-06T02:41:37.370 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-06T02:41:37.371 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.max_total_wal_size: 0 2024-04-06T02:41:37.371 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-06T02:41:37.371 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-06T02:41:37.371 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-06T02:41:37.371 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-06T02:41:37.371 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.max_open_files: -1 2024-04-06T02:41:37.371 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.bytes_per_sync: 0 2024-04-06T02:41:37.371 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-06T02:41:37.371 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-06T02:41:37.371 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.compaction_readahead_size: 0 2024-04-06T02:41:37.371 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.max_background_flushes: -1 2024-04-06T02:41:37.371 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Compression algorithms supported: 2024-04-06T02:41:37.371 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-06T02:41:37.372 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: kZSTD supported: 0 2024-04-06T02:41:37.372 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: kXpressCompression supported: 0 2024-04-06T02:41:37.372 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: kLZ4HCCompression supported: 1 2024-04-06T02:41:37.373 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: kZlibCompression supported: 1 2024-04-06T02:41:37.373 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: kSnappyCompression supported: 1 2024-04-06T02:41:37.373 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: kLZ4Compression supported: 1 2024-04-06T02:41:37.373 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: kBZip2Compression supported: 0 2024-04-06T02:41:37.373 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-06T02:41:37.373 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-06T02:41:37.373 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi105/store.db/MANIFEST-000010 2024-04-06T02:41:37.373 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-06T02:41:37.373 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-06T02:41:37.373 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.merge_operator: 2024-04-06T02:41:37.373 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.compaction_filter: None 2024-04-06T02:41:37.374 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.compaction_filter_factory: None 2024-04-06T02:41:37.374 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.sst_partitioner_factory: None 2024-04-06T02:41:37.374 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-06T02:41:37.374 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-06T02:41:37.374 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5566f85d53e0) 2024-04-06T02:41:37.374 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: cache_index_and_filter_blocks: 1 2024-04-06T02:41:37.374 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-06T02:41:37.374 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-06T02:41:37.374 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: pin_top_level_index_and_filter: 1 2024-04-06T02:41:37.374 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: index_type: 0 2024-04-06T02:41:37.374 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: data_block_index_type: 0 2024-04-06T02:41:37.375 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: index_shortening: 1 2024-04-06T02:41:37.375 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-06T02:41:37.375 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: checksum: 4 2024-04-06T02:41:37.375 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: no_block_cache: 0 2024-04-06T02:41:37.375 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: block_cache: 0x5566f78e7350 2024-04-06T02:41:37.375 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: block_cache_name: BinnedLRUCache 2024-04-06T02:41:37.375 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: block_cache_options: 2024-04-06T02:41:37.375 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: capacity : 536870912 2024-04-06T02:41:37.375 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: num_shard_bits : 4 2024-04-06T02:41:37.375 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: strict_capacity_limit : 0 2024-04-06T02:41:37.376 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: high_pri_pool_ratio: 0.000 2024-04-06T02:41:37.376 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: block_cache_compressed: (nil) 2024-04-06T02:41:37.376 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: persistent_cache: (nil) 2024-04-06T02:41:37.376 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: block_size: 4096 2024-04-06T02:41:37.376 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: block_size_deviation: 10 2024-04-06T02:41:37.376 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: block_restart_interval: 16 2024-04-06T02:41:37.376 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: index_block_restart_interval: 1 2024-04-06T02:41:37.376 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: metadata_block_size: 4096 2024-04-06T02:41:37.376 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: partition_filters: 0 2024-04-06T02:41:37.377 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: use_delta_encoding: 1 2024-04-06T02:41:37.377 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: filter_policy: bloomfilter 2024-04-06T02:41:37.377 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: whole_key_filtering: 1 2024-04-06T02:41:37.377 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: verify_compression: 0 2024-04-06T02:41:37.377 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: read_amp_bytes_per_bit: 0 2024-04-06T02:41:37.377 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: format_version: 5 2024-04-06T02:41:37.377 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: enable_index_compression: 1 2024-04-06T02:41:37.377 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: block_align: 0 2024-04-06T02:41:37.377 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: max_auto_readahead_size: 262144 2024-04-06T02:41:37.377 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: prepopulate_block_cache: 0 2024-04-06T02:41:37.377 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: initial_auto_readahead_size: 8192 2024-04-06T02:41:37.377 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: num_file_reads_for_auto_readahead: 2 2024-04-06T02:41:37.378 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.write_buffer_size: 33554432 2024-04-06T02:41:37.378 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.max_write_buffer_number: 2 2024-04-06T02:41:37.378 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.compression: NoCompression 2024-04-06T02:41:37.378 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.bottommost_compression: Disabled 2024-04-06T02:41:37.378 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.prefix_extractor: nullptr 2024-04-06T02:41:37.378 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-06T02:41:37.378 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.num_levels: 7 2024-04-06T02:41:37.378 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-06T02:41:37.378 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-06T02:41:37.378 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-06T02:41:37.378 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-06T02:41:37.378 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-06T02:41:37.378 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-06T02:41:37.379 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-06T02:41:37.379 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-06T02:41:37.379 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-06T02:41:37.379 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-06T02:41:37.379 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-06T02:41:37.379 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-06T02:41:37.379 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-06T02:41:37.379 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.compression_opts.level: 32767 2024-04-06T02:41:37.379 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.compression_opts.strategy: 0 2024-04-06T02:41:37.379 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-06T02:41:37.379 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-06T02:41:37.379 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-06T02:41:37.379 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-06T02:41:37.379 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.compression_opts.enabled: false 2024-04-06T02:41:37.380 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-06T02:41:37.380 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-06T02:41:37.380 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-06T02:41:37.380 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-06T02:41:37.380 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.target_file_size_base: 67108864 2024-04-06T02:41:37.380 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-06T02:41:37.380 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-06T02:41:37.380 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-06T02:41:37.380 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-06T02:41:37.380 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-06T02:41:37.380 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-06T02:41:37.380 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-06T02:41:37.381 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-06T02:41:37.381 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-06T02:41:37.381 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-06T02:41:37.381 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-06T02:41:37.381 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-06T02:41:37.381 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-06T02:41:37.382 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-06T02:41:37.382 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.arena_block_size: 1048576 2024-04-06T02:41:37.382 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-06T02:41:37.382 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-06T02:41:37.382 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.disable_auto_compactions: 0 2024-04-06T02:41:37.382 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-06T02:41:37.382 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-06T02:41:37.382 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-06T02:41:37.383 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-06T02:41:37.383 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-06T02:41:37.383 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-06T02:41:37.383 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-06T02:41:37.383 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-06T02:41:37.383 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-06T02:41:37.383 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-06T02:41:37.383 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.table_properties_collectors: 2024-04-06T02:41:37.383 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.inplace_update_support: 0 2024-04-06T02:41:37.383 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-06T02:41:37.383 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-06T02:41:37.384 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-06T02:41:37.384 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-06T02:41:37.384 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.bloom_locality: 0 2024-04-06T02:41:37.384 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.max_successive_merges: 0 2024-04-06T02:41:37.384 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-06T02:41:37.384 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.paranoid_file_checks: 0 2024-04-06T02:41:37.384 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.force_consistency_checks: 1 2024-04-06T02:41:37.384 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.report_bg_io_stats: 0 2024-04-06T02:41:37.384 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.ttl: 2592000 2024-04-06T02:41:37.384 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-06T02:41:37.384 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-06T02:41:37.384 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-06T02:41:37.384 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.enable_blob_files: false 2024-04-06T02:41:37.385 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.min_blob_size: 0 2024-04-06T02:41:37.385 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.blob_file_size: 268435456 2024-04-06T02:41:37.385 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-06T02:41:37.385 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-06T02:41:37.385 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-06T02:41:37.385 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-06T02:41:37.385 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-06T02:41:37.385 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.blob_file_starting_level: 0 2024-04-06T02:41:37.385 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-06T02:41:37.385 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi105/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-06T02:41:37.385 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-06T02:41:37.385 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: af7e5e27-4532-4ea9-9dc8-1c1fe9a615b1 2024-04-06T02:41:37.386 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712371297280044, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-06T02:41:37.386 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-06T02:41:37.386 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712371297281077, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93627, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 91649, "index_size": 260, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13576, "raw_average_key_size": 49, "raw_value_size": 84258, "raw_average_value_size": 308, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712371297, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "af7e5e27-4532-4ea9-9dc8-1c1fe9a615b1", "db_session_id": "KYBD78FR1RLA65PXSTHK", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-06T02:41:37.386 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712371297281158, "job": 1, "event": "recovery_finished"} 2024-04-06T02:41:37.386 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-06T02:41:37.386 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi105/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-06T02:41:37.386 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5566f7984000 2024-04-06T02:41:37.386 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: DB pointer 0x5566f7970000 2024-04-06T02:41:37.386 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-06T02:41:37.386 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-06T02:41:37.386 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: ** DB Stats ** 2024-04-06T02:41:37.387 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T02:41:37.387 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-06T02:41:37.387 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T02:41:37.387 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T02:41:37.387 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-06T02:41:37.387 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T02:41:37.387 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T02:41:37.387 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: 2024-04-06T02:41:37.387 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: ** Compaction Stats [default] ** 2024-04-06T02:41:37.387 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-06T02:41:37.387 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-06T02:41:37.387 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: L0 2/0 93.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 132.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-06T02:41:37.388 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Sum 2/0 93.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 132.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-06T02:41:37.388 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 132.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-06T02:41:37.388 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: 2024-04-06T02:41:37.388 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: ** Compaction Stats [default] ** 2024-04-06T02:41:37.388 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-06T02:41:37.388 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T02:41:37.388 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 132.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-06T02:41:37.388 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: 2024-04-06T02:41:37.388 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-06T02:41:37.388 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: 2024-04-06T02:41:37.388 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T02:41:37.388 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-06T02:41:37.388 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-06T02:41:37.389 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-06T02:41:37.389 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-06T02:41:37.389 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-06T02:41:37.389 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Cumulative compaction: 0.00 GB write, 30.16 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T02:41:37.389 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Interval compaction: 0.00 GB write, 30.16 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T02:41:37.389 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-06T02:41:37.389 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Block cache BinnedLRUCache@0x5566f78e7350#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-04-06T02:41:37.389 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-04-06T02:41:37.389 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: 2024-04-06T02:41:37.389 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-06T02:41:37.389 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: starting mon.smithi105 rank 0 at public addrs [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] at bind addrs [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi105 fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 2024-04-06T02:41:37.389 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: mon.smithi105@-1(???) e1 preinit fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 2024-04-06T02:41:37.389 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: mon.smithi105@-1(???).mds e1 new map 2024-04-06T02:41:37.390 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: mon.smithi105@-1(???).mds e1 print_map 2024-04-06T02:41:37.390 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: e1 2024-04-06T02:41:37.390 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T02:41:37.390 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T02:41:37.390 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: legacy client fscid: -1 2024-04-06T02:41:37.390 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: 2024-04-06T02:41:37.390 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: No filesystems configured 2024-04-06T02:41:37.390 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: mon.smithi105@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-06T02:41:37.390 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: mon.smithi105@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-06T02:41:37.390 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: mon.smithi105@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-06T02:41:37.391 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: mon.smithi105@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-06T02:41:37.391 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: mon.smithi105@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-06T02:41:37.391 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: expand_channel_meta expand map: {default=false} 2024-04-06T02:41:37.391 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: expand_channel_meta from 'false' to 'false' 2024-04-06T02:41:37.391 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: expand_channel_meta expanded map: {default=false} 2024-04-06T02:41:37.391 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: expand_channel_meta expand map: {default=info} 2024-04-06T02:41:37.391 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: expand_channel_meta from 'info' to 'info' 2024-04-06T02:41:37.391 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: expand_channel_meta expanded map: {default=info} 2024-04-06T02:41:37.391 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: expand_channel_meta expand map: {default=daemon} 2024-04-06T02:41:37.391 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: expand_channel_meta from 'daemon' to 'daemon' 2024-04-06T02:41:37.391 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: expand_channel_meta expanded map: {default=daemon} 2024-04-06T02:41:37.391 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: expand_channel_meta expand map: {default=debug} 2024-04-06T02:41:37.392 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: expand_channel_meta from 'debug' to 'debug' 2024-04-06T02:41:37.392 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: expand_channel_meta expanded map: {default=debug} 2024-04-06T02:41:37.392 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 bash[29209]: d27bf0498159c5eb042ec29dc651c2d3cde19f6c07dbc359b8c9870c0e665bda 2024-04-06T02:41:37.392 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 systemd[1]: Started Ceph mon.smithi105 for 130df578-f3bf-11ee-b649-cb9ed24678a4. 2024-04-06T02:41:37.392 INFO:teuthology.orchestra.run.smithi105.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-06T02:41:37.643 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: mon.smithi105 is new leader, mons smithi105 in quorum (ranks 0) 2024-04-06T02:41:37.643 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: monmap e1: 1 mons at {smithi105=[v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-06T02:41:37.643 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: fsmap 2024-04-06T02:41:37.643 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: osdmap e1: 0 total, 0 up, 0 in 2024-04-06T02:41:37.643 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:37 smithi105 ceph-mon[29228]: mgrmap e1: no daemons active 2024-04-06T02:41:38.381 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.759+0000 7f9c899df700 1 Processor -- start 2024-04-06T02:41:38.382 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.759+0000 7f9c899df700 1 -- start start 2024-04-06T02:41:38.382 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.760+0000 7f9c899df700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c841064e0 0x7f9c84106900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:41:38.382 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.760+0000 7f9c899df700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c84106ed0 con 0x7f9c841064e0 2024-04-06T02:41:38.382 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.760+0000 7f9c82ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c841064e0 0x7f9c84106900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:41:38.382 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.760+0000 7f9c82ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c841064e0 0x7f9c84106900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33930/0 (socket says 172.21.15.105:33930) 2024-04-06T02:41:38.382 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.760+0000 7f9c82ffd700 1 -- 172.21.15.105:0/3302444027 learned_addr learned my addr 172.21.15.105:0/3302444027 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:41:38.382 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.760+0000 7f9c82ffd700 1 -- 172.21.15.105:0/3302444027 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c841076e0 con 0x7f9c841064e0 2024-04-06T02:41:38.382 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.760+0000 7f9c82ffd700 1 --2- 172.21.15.105:0/3302444027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c841064e0 0x7f9c84106900 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9c780098a0 tx=0x7f9c78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d7a8a153a3cc88 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:41:38.383 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.761+0000 7f9c81ffb700 1 -- 172.21.15.105:0/3302444027 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c78004180 con 0x7f9c841064e0 2024-04-06T02:41:38.383 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.761+0000 7f9c81ffb700 1 -- 172.21.15.105:0/3302444027 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9c780042e0 con 0x7f9c841064e0 2024-04-06T02:41:38.383 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.761+0000 7f9c81ffb700 1 -- 172.21.15.105:0/3302444027 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c78011a50 con 0x7f9c841064e0 2024-04-06T02:41:38.383 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.761+0000 7f9c899df700 1 -- 172.21.15.105:0/3302444027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c841064e0 msgr2=0x7f9c84106900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:41:38.383 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.761+0000 7f9c899df700 1 --2- 172.21.15.105:0/3302444027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c841064e0 0x7f9c84106900 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9c780098a0 tx=0x7f9c78009bb0 comp rx=0 tx=0).stop 2024-04-06T02:41:38.383 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.761+0000 7f9c899df700 1 -- 172.21.15.105:0/3302444027 shutdown_connections 2024-04-06T02:41:38.383 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.761+0000 7f9c899df700 1 --2- 172.21.15.105:0/3302444027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c841064e0 0x7f9c84106900 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:41:38.383 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.761+0000 7f9c899df700 1 -- 172.21.15.105:0/3302444027 >> 172.21.15.105:0/3302444027 conn(0x7f9c84101a80 msgr2=0x7f9c84103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:41:38.383 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.761+0000 7f9c899df700 1 -- 172.21.15.105:0/3302444027 shutdown_connections 2024-04-06T02:41:38.384 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.761+0000 7f9c899df700 1 -- 172.21.15.105:0/3302444027 wait complete. 2024-04-06T02:41:38.384 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.762+0000 7f9c899df700 1 Processor -- start 2024-04-06T02:41:38.384 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.762+0000 7f9c899df700 1 -- start start 2024-04-06T02:41:38.384 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.762+0000 7f9c899df700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c841064e0 0x7f9c8419a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:41:38.384 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.762+0000 7f9c899df700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8419a690 con 0x7f9c841064e0 2024-04-06T02:41:38.384 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.762+0000 7f9c82ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c841064e0 0x7f9c8419a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:41:38.384 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.763+0000 7f9c82ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c841064e0 0x7f9c8419a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33936/0 (socket says 172.21.15.105:33936) 2024-04-06T02:41:38.384 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.763+0000 7f9c82ffd700 1 -- 172.21.15.105:0/3834508250 learned_addr learned my addr 172.21.15.105:0/3834508250 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:41:38.385 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.763+0000 7f9c82ffd700 1 -- 172.21.15.105:0/3834508250 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c78009550 con 0x7f9c841064e0 2024-04-06T02:41:38.385 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.763+0000 7f9c82ffd700 1 --2- 172.21.15.105:0/3834508250 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c841064e0 0x7f9c8419a150 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9c78006af0 tx=0x7f9c78011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:41:38.385 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.763+0000 7f9c889dd700 1 -- 172.21.15.105:0/3834508250 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c78003b90 con 0x7f9c841064e0 2024-04-06T02:41:38.385 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.763+0000 7f9c889dd700 1 -- 172.21.15.105:0/3834508250 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9c7801a8a0 con 0x7f9c841064e0 2024-04-06T02:41:38.385 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.763+0000 7f9c889dd700 1 -- 172.21.15.105:0/3834508250 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c78019880 con 0x7f9c841064e0 2024-04-06T02:41:38.385 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.763+0000 7f9c899df700 1 -- 172.21.15.105:0/3834508250 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c8419a890 con 0x7f9c841064e0 2024-04-06T02:41:38.385 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.763+0000 7f9c899df700 1 -- 172.21.15.105:0/3834508250 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c8419ad30 con 0x7f9c841064e0 2024-04-06T02:41:38.385 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.763+0000 7f9c889dd700 1 -- 172.21.15.105:0/3834508250 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9c7802d440 con 0x7f9c841064e0 2024-04-06T02:41:38.385 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.763+0000 7f9c889dd700 1 -- 172.21.15.105:0/3834508250 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9c78024800 con 0x7f9c841064e0 2024-04-06T02:41:38.386 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.764+0000 7f9c899df700 1 -- 172.21.15.105:0/3834508250 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c840623c0 con 0x7f9c841064e0 2024-04-06T02:41:38.386 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.766+0000 7f9c889dd700 1 -- 172.21.15.105:0/3834508250 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f9c7801aa10 con 0x7f9c841064e0 2024-04-06T02:41:38.386 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.832+0000 7f9c899df700 1 -- 172.21.15.105:0/3834508250 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f9c8410a990 con 0x7f9c841064e0 2024-04-06T02:41:38.386 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.837+0000 7f9c889dd700 1 -- 172.21.15.105:0/3834508250 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f9c7801abb0 con 0x7f9c841064e0 2024-04-06T02:41:38.386 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.838+0000 7f9c899df700 1 -- 172.21.15.105:0/3834508250 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c841064e0 msgr2=0x7f9c8419a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:41:38.386 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.838+0000 7f9c899df700 1 --2- 172.21.15.105:0/3834508250 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c841064e0 0x7f9c8419a150 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9c78006af0 tx=0x7f9c78011a90 comp rx=0 tx=0).stop 2024-04-06T02:41:38.387 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.838+0000 7f9c899df700 1 -- 172.21.15.105:0/3834508250 shutdown_connections 2024-04-06T02:41:38.387 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.838+0000 7f9c899df700 1 --2- 172.21.15.105:0/3834508250 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c841064e0 0x7f9c8419a150 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:41:38.387 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.838+0000 7f9c899df700 1 -- 172.21.15.105:0/3834508250 >> 172.21.15.105:0/3834508250 conn(0x7f9c84101a80 msgr2=0x7f9c84102800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:41:38.387 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.838+0000 7f9c899df700 1 -- 172.21.15.105:0/3834508250 shutdown_connections 2024-04-06T02:41:38.387 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:37.838+0000 7f9c899df700 1 -- 172.21.15.105:0/3834508250 wait complete. 2024-04-06T02:41:38.388 INFO:teuthology.orchestra.run.smithi105.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-06T02:41:38.397 INFO:teuthology.orchestra.run.smithi105.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-06T02:41:38.397 INFO:teuthology.orchestra.run.smithi105.stdout:Creating mgr... 2024-04-06T02:41:38.397 INFO:teuthology.orchestra.run.smithi105.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-06T02:41:38.398 INFO:teuthology.orchestra.run.smithi105.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-06T02:41:38.398 INFO:teuthology.orchestra.run.smithi105.stdout:Verifying port 0.0.0.0:8443 ... 2024-04-06T02:41:38.644 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-130df578-f3bf-11ee-b649-cb9ed24678a4@mgr.smithi105.vqqcpx 2024-04-06T02:41:38.644 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Failed to reset failed state of unit ceph-130df578-f3bf-11ee-b649-cb9ed24678a4@mgr.smithi105.vqqcpx.service: Unit ceph-130df578-f3bf-11ee-b649-cb9ed24678a4@mgr.smithi105.vqqcpx.service not loaded. 2024-04-06T02:41:38.787 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-130df578-f3bf-11ee-b649-cb9ed24678a4.target.wants/ceph-130df578-f3bf-11ee-b649-cb9ed24678a4@mgr.smithi105.vqqcpx.service → /etc/systemd/system/ceph-130df578-f3bf-11ee-b649-cb9ed24678a4@.service. 2024-04-06T02:41:38.923 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:38 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/3834508250' entity='client.admin' 2024-04-06T02:41:39.349 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-06T02:41:39.350 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout disabled 2024-04-06T02:41:39.354 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-06T02:41:39.354 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout inactive 2024-04-06T02:41:39.354 INFO:teuthology.orchestra.run.smithi105.stdout:firewalld.service is not enabled 2024-04-06T02:41:39.354 INFO:teuthology.orchestra.run.smithi105.stdout:Not possible to enable service . firewalld.service is not available 2024-04-06T02:41:39.362 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-06T02:41:39.362 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout disabled 2024-04-06T02:41:39.369 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-06T02:41:39.370 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout inactive 2024-04-06T02:41:39.370 INFO:teuthology.orchestra.run.smithi105.stdout:firewalld.service is not enabled 2024-04-06T02:41:39.370 INFO:teuthology.orchestra.run.smithi105.stdout:Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2024-04-06T02:41:39.370 INFO:teuthology.orchestra.run.smithi105.stdout:Waiting for mgr to start... 2024-04-06T02:41:39.370 INFO:teuthology.orchestra.run.smithi105.stdout:Waiting for mgr... 2024-04-06T02:41:40.173 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:39 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/1338925641' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-06T02:41:40.447 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-04-06T02:41:40.447 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout { 2024-04-06T02:41:40.447 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "fsid": "130df578-f3bf-11ee-b649-cb9ed24678a4", 2024-04-06T02:41:40.447 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "health": { 2024-04-06T02:41:40.447 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-06T02:41:40.447 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-06T02:41:40.447 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-06T02:41:40.447 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-06T02:41:40.448 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-06T02:41:40.448 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-06T02:41:40.448 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 0 2024-04-06T02:41:40.448 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-04-06T02:41:40.448 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-06T02:41:40.448 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "smithi105" 2024-04-06T02:41:40.448 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-04-06T02:41:40.448 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-06T02:41:40.448 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-06T02:41:40.448 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T02:41:40.448 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-06T02:41:40.448 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-06T02:41:40.448 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-06T02:41:40.448 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-06T02:41:40.449 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T02:41:40.449 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-06T02:41:40.449 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-06T02:41:40.449 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-06T02:41:40.457 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-06T02:41:40.457 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-06T02:41:40.457 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-06T02:41:40.457 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-06T02:41:40.457 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-06T02:41:40.457 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-06T02:41:40.458 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-06T02:41:40.458 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-06T02:41:40.458 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-06T02:41:40.458 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-06T02:41:40.458 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-06T02:41:40.458 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-06T02:41:40.458 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-06T02:41:40.458 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-06T02:41:40.458 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-06T02:41:40.458 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T02:41:40.458 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-06T02:41:40.458 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-06T02:41:40.458 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-06T02:41:40.459 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-06T02:41:40.459 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-06T02:41:40.459 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-06T02:41:40.459 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-06T02:41:40.459 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-06T02:41:40.459 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-06T02:41:40.459 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "restful" 2024-04-06T02:41:40.459 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-04-06T02:41:40.459 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-06T02:41:40.459 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-06T02:41:40.459 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-06T02:41:40.459 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T02:41:40.459 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "modified": "2024-04-06T02:41:32.645448+0000", 2024-04-06T02:41:40.459 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-06T02:41:40.460 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-06T02:41:40.460 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-06T02:41:40.460 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout } 2024-04-06T02:41:40.460 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.794+0000 7fd0aca4a700 1 Processor -- start 2024-04-06T02:41:40.460 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.794+0000 7fd0aca4a700 1 -- start start 2024-04-06T02:41:40.460 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.794+0000 7fd0aca4a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0a8106a50 0x7fd0a8106e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:41:40.460 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.794+0000 7fd0aca4a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a8107440 con 0x7fd0a8106a50 2024-04-06T02:41:40.460 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.794+0000 7fd09e59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0a8106a50 0x7fd0a8106e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:41:40.460 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.794+0000 7fd09e59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0a8106a50 0x7fd0a8106e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33968/0 (socket says 172.21.15.105:33968) 2024-04-06T02:41:40.460 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.794+0000 7fd09e59c700 1 -- 172.21.15.105:0/4080132769 learned_addr learned my addr 172.21.15.105:0/4080132769 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:41:40.460 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.794+0000 7fd09e59c700 1 -- 172.21.15.105:0/4080132769 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a8105660 con 0x7fd0a8106a50 2024-04-06T02:41:40.460 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.795+0000 7fd09e59c700 1 --2- 172.21.15.105:0/4080132769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0a8106a50 0x7fd0a8106e70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd0980098a0 tx=0x7fd098009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3b4a96ee4ef8c9c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:41:40.460 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.795+0000 7fd0a5d9b700 1 -- 172.21.15.105:0/4080132769 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd098004180 con 0x7fd0a8106a50 2024-04-06T02:41:40.460 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.795+0000 7fd0a5d9b700 1 -- 172.21.15.105:0/4080132769 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd0980042e0 con 0x7fd0a8106a50 2024-04-06T02:41:40.461 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.795+0000 7fd0a5d9b700 1 -- 172.21.15.105:0/4080132769 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd098011a50 con 0x7fd0a8106a50 2024-04-06T02:41:40.461 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.795+0000 7fd0aca4a700 1 -- 172.21.15.105:0/4080132769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0a8106a50 msgr2=0x7fd0a8106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:41:40.461 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.795+0000 7fd0aca4a700 1 --2- 172.21.15.105:0/4080132769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0a8106a50 0x7fd0a8106e70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd0980098a0 tx=0x7fd098009bb0 comp rx=0 tx=0).stop 2024-04-06T02:41:40.461 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.795+0000 7fd0aca4a700 1 -- 172.21.15.105:0/4080132769 shutdown_connections 2024-04-06T02:41:40.461 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.795+0000 7fd0aca4a700 1 --2- 172.21.15.105:0/4080132769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0a8106a50 0x7fd0a8106e70 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:41:40.461 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.795+0000 7fd0aca4a700 1 -- 172.21.15.105:0/4080132769 >> 172.21.15.105:0/4080132769 conn(0x7fd0a8100190 msgr2=0x7fd0a81025d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:41:40.461 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.796+0000 7fd0aca4a700 1 -- 172.21.15.105:0/4080132769 shutdown_connections 2024-04-06T02:41:40.461 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.796+0000 7fd0aca4a700 1 -- 172.21.15.105:0/4080132769 wait complete. 2024-04-06T02:41:40.461 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.796+0000 7fd0aca4a700 1 Processor -- start 2024-04-06T02:41:40.461 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.796+0000 7fd0aca4a700 1 -- start start 2024-04-06T02:41:40.461 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.797+0000 7fd0aca4a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0a8106a50 0x7fd0a8105110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:41:40.461 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.797+0000 7fd0aca4a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a819ceb0 con 0x7fd0a8106a50 2024-04-06T02:41:40.461 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.797+0000 7fd09e59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0a8106a50 0x7fd0a8105110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:41:40.462 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.797+0000 7fd09e59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0a8106a50 0x7fd0a8105110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33984/0 (socket says 172.21.15.105:33984) 2024-04-06T02:41:40.462 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.797+0000 7fd09e59c700 1 -- 172.21.15.105:0/1338925641 learned_addr learned my addr 172.21.15.105:0/1338925641 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:41:40.462 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.797+0000 7fd09e59c700 1 -- 172.21.15.105:0/1338925641 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd098009550 con 0x7fd0a8106a50 2024-04-06T02:41:40.462 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.797+0000 7fd09e59c700 1 --2- 172.21.15.105:0/1338925641 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0a8106a50 0x7fd0a8105110 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd098000c00 tx=0x7fd098011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:41:40.462 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.797+0000 7fd09ffff700 1 -- 172.21.15.105:0/1338925641 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd098003b90 con 0x7fd0a8106a50 2024-04-06T02:41:40.462 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.797+0000 7fd0aca4a700 1 -- 172.21.15.105:0/1338925641 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0a819cff0 con 0x7fd0a8106a50 2024-04-06T02:41:40.462 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.797+0000 7fd09ffff700 1 -- 172.21.15.105:0/1338925641 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd0980248b0 con 0x7fd0a8106a50 2024-04-06T02:41:40.462 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.797+0000 7fd09ffff700 1 -- 172.21.15.105:0/1338925641 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd09801a8e0 con 0x7fd0a8106a50 2024-04-06T02:41:40.462 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.797+0000 7fd0aca4a700 1 -- 172.21.15.105:0/1338925641 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0a819d490 con 0x7fd0a8106a50 2024-04-06T02:41:40.462 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.798+0000 7fd09ffff700 1 -- 172.21.15.105:0/1338925641 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd09802d3f0 con 0x7fd0a8106a50 2024-04-06T02:41:40.463 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.798+0000 7fd09ffff700 1 -- 172.21.15.105:0/1338925641 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd098019490 con 0x7fd0a8106a50 2024-04-06T02:41:40.463 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.798+0000 7fd0aca4a700 1 -- 172.21.15.105:0/1338925641 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0a80623c0 con 0x7fd0a8106a50 2024-04-06T02:41:40.463 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.801+0000 7fd09ffff700 1 -- 172.21.15.105:0/1338925641 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fd098024a20 con 0x7fd0a8106a50 2024-04-06T02:41:40.463 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.872+0000 7fd0aca4a700 1 -- 172.21.15.105:0/1338925641 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd0a819daf0 con 0x7fd0a8106a50 2024-04-06T02:41:40.463 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.872+0000 7fd09ffff700 1 -- 172.21.15.105:0/1338925641 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7fd098035090 con 0x7fd0a8106a50 2024-04-06T02:41:40.463 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.873+0000 7fd0aca4a700 1 -- 172.21.15.105:0/1338925641 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0a8106a50 msgr2=0x7fd0a8105110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:41:40.463 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.873+0000 7fd0aca4a700 1 --2- 172.21.15.105:0/1338925641 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0a8106a50 0x7fd0a8105110 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd098000c00 tx=0x7fd098011ab0 comp rx=0 tx=0).stop 2024-04-06T02:41:40.463 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.873+0000 7fd0aca4a700 1 -- 172.21.15.105:0/1338925641 shutdown_connections 2024-04-06T02:41:40.463 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.873+0000 7fd0aca4a700 1 --2- 172.21.15.105:0/1338925641 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0a8106a50 0x7fd0a8105110 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:41:40.463 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.873+0000 7fd0aca4a700 1 -- 172.21.15.105:0/1338925641 >> 172.21.15.105:0/1338925641 conn(0x7fd0a8100190 msgr2=0x7fd0a8100ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:41:40.464 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.873+0000 7fd0aca4a700 1 -- 172.21.15.105:0/1338925641 shutdown_connections 2024-04-06T02:41:40.464 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:39.873+0000 7fd0aca4a700 1 -- 172.21.15.105:0/1338925641 wait complete. 2024-04-06T02:41:40.464 INFO:teuthology.orchestra.run.smithi105.stdout:mgr not available, waiting (1/15)... 2024-04-06T02:41:44.423 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:44 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/3040918122' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-06T02:41:46.173 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:45 smithi105 ceph-mon[29228]: Activating manager daemon smithi105.vqqcpx 2024-04-06T02:41:46.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:45 smithi105 ceph-mon[29228]: mgrmap e2: smithi105.vqqcpx(active, starting, since 0.00313609s) 2024-04-06T02:41:46.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:45 smithi105 ceph-mon[29228]: from='mgr.14100 172.21.15.105:0/1858465477' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T02:41:46.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:45 smithi105 ceph-mon[29228]: from='mgr.14100 172.21.15.105:0/1858465477' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T02:41:46.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:45 smithi105 ceph-mon[29228]: from='mgr.14100 172.21.15.105:0/1858465477' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T02:41:46.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:45 smithi105 ceph-mon[29228]: from='mgr.14100 172.21.15.105:0/1858465477' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-04-06T02:41:46.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:45 smithi105 ceph-mon[29228]: from='mgr.14100 172.21.15.105:0/1858465477' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mgr metadata", "who": "smithi105.vqqcpx", "id": "smithi105.vqqcpx"}]: dispatch 2024-04-06T02:41:46.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:45 smithi105 ceph-mon[29228]: Manager daemon smithi105.vqqcpx is now available 2024-04-06T02:41:46.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:45 smithi105 ceph-mon[29228]: from='mgr.14100 172.21.15.105:0/1858465477' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi105.vqqcpx/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:41:46.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:45 smithi105 ceph-mon[29228]: from='mgr.14100 172.21.15.105:0/1858465477' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi105.vqqcpx/trash_purge_schedule"}]: dispatch 2024-04-06T02:41:46.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:45 smithi105 ceph-mon[29228]: from='mgr.14100 172.21.15.105:0/1858465477' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:41:46.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:45 smithi105 ceph-mon[29228]: from='mgr.14100 172.21.15.105:0/1858465477' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:41:46.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:45 smithi105 ceph-mon[29228]: from='mgr.14100 172.21.15.105:0/1858465477' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:41:48.173 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:47 smithi105 ceph-mon[29228]: mgrmap e3: smithi105.vqqcpx(active, since 1.00717s) 2024-04-06T02:41:50.173 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:49 smithi105 ceph-mon[29228]: mgrmap e4: smithi105.vqqcpx(active, since 3s) 2024-04-06T02:41:50.300 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-04-06T02:41:50.300 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout { 2024-04-06T02:41:50.300 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "fsid": "130df578-f3bf-11ee-b649-cb9ed24678a4", 2024-04-06T02:41:50.300 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "health": { 2024-04-06T02:41:50.300 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-06T02:41:50.300 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-06T02:41:50.300 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-06T02:41:50.300 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-06T02:41:50.300 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-06T02:41:50.300 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-06T02:41:50.301 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 0 2024-04-06T02:41:50.301 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-04-06T02:41:50.301 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-06T02:41:50.301 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "smithi105" 2024-04-06T02:41:50.301 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-04-06T02:41:50.301 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-04-06T02:41:50.301 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-06T02:41:50.301 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T02:41:50.301 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-06T02:41:50.301 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-06T02:41:50.301 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-06T02:41:50.301 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-06T02:41:50.301 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T02:41:50.301 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-06T02:41:50.302 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-06T02:41:50.302 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-06T02:41:50.302 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-06T02:41:50.302 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-06T02:41:50.302 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-06T02:41:50.302 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-06T02:41:50.302 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-06T02:41:50.302 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-06T02:41:50.302 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-06T02:41:50.302 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-06T02:41:50.302 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-06T02:41:50.302 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-06T02:41:50.302 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-06T02:41:50.302 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-06T02:41:50.303 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-06T02:41:50.303 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-06T02:41:50.303 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-06T02:41:50.303 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T02:41:50.303 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-06T02:41:50.303 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-06T02:41:50.303 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-06T02:41:50.303 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-06T02:41:50.303 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-06T02:41:50.303 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-06T02:41:50.303 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-06T02:41:50.303 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-06T02:41:50.303 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-06T02:41:50.303 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "restful" 2024-04-06T02:41:50.304 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-04-06T02:41:50.304 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-06T02:41:50.304 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-06T02:41:50.304 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-06T02:41:50.304 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T02:41:50.304 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "modified": "2024-04-06T02:41:32.645448+0000", 2024-04-06T02:41:50.304 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-06T02:41:50.304 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-06T02:41:50.304 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-06T02:41:50.304 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout } 2024-04-06T02:41:50.304 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.038+0000 7fa24851d700 1 Processor -- start 2024-04-06T02:41:50.304 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.038+0000 7fa24851d700 1 -- start start 2024-04-06T02:41:50.304 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.039+0000 7fa24851d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa240105470 0x7fa240105890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:41:50.305 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.039+0000 7fa24851d700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa240105e60 con 0x7fa240105470 2024-04-06T02:41:50.305 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.039+0000 7fa2462b9700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa240105470 0x7fa240105890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:41:50.305 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.039+0000 7fa2462b9700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa240105470 0x7fa240105890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:39528/0 (socket says 172.21.15.105:39528) 2024-04-06T02:41:50.305 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.039+0000 7fa2462b9700 1 -- 172.21.15.105:0/2715730819 learned_addr learned my addr 172.21.15.105:0/2715730819 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:41:50.305 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.039+0000 7fa2462b9700 1 -- 172.21.15.105:0/2715730819 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa240106670 con 0x7fa240105470 2024-04-06T02:41:50.305 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.039+0000 7fa2462b9700 1 --2- 172.21.15.105:0/2715730819 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa240105470 0x7fa240105890 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa23c0098a0 tx=0x7fa23c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77a817ba1abcccbf server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:41:50.305 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.039+0000 7fa2452b7700 1 -- 172.21.15.105:0/2715730819 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa23c004180 con 0x7fa240105470 2024-04-06T02:41:50.305 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.040+0000 7fa2452b7700 1 -- 172.21.15.105:0/2715730819 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa23c0042e0 con 0x7fa240105470 2024-04-06T02:41:50.305 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.040+0000 7fa2452b7700 1 -- 172.21.15.105:0/2715730819 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa23c011a50 con 0x7fa240105470 2024-04-06T02:41:50.305 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.040+0000 7fa24851d700 1 -- 172.21.15.105:0/2715730819 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa240105470 msgr2=0x7fa240105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:41:50.305 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.040+0000 7fa24851d700 1 --2- 172.21.15.105:0/2715730819 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa240105470 0x7fa240105890 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa23c0098a0 tx=0x7fa23c009bb0 comp rx=0 tx=0).stop 2024-04-06T02:41:50.305 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.040+0000 7fa24851d700 1 -- 172.21.15.105:0/2715730819 shutdown_connections 2024-04-06T02:41:50.305 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.040+0000 7fa24851d700 1 --2- 172.21.15.105:0/2715730819 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa240105470 0x7fa240105890 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:41:50.306 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.040+0000 7fa24851d700 1 -- 172.21.15.105:0/2715730819 >> 172.21.15.105:0/2715730819 conn(0x7fa2401009f0 msgr2=0x7fa240102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:41:50.306 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.040+0000 7fa24851d700 1 -- 172.21.15.105:0/2715730819 shutdown_connections 2024-04-06T02:41:50.306 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.040+0000 7fa24851d700 1 -- 172.21.15.105:0/2715730819 wait complete. 2024-04-06T02:41:50.306 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.041+0000 7fa24851d700 1 Processor -- start 2024-04-06T02:41:50.306 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.041+0000 7fa24851d700 1 -- start start 2024-04-06T02:41:50.306 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.041+0000 7fa24851d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa240105470 0x7fa240199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:41:50.306 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.041+0000 7fa24851d700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2401995d0 con 0x7fa240105470 2024-04-06T02:41:50.306 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.042+0000 7fa2462b9700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa240105470 0x7fa240199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:41:50.306 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.042+0000 7fa2462b9700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa240105470 0x7fa240199090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:39542/0 (socket says 172.21.15.105:39542) 2024-04-06T02:41:50.306 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.042+0000 7fa2462b9700 1 -- 172.21.15.105:0/3040918122 learned_addr learned my addr 172.21.15.105:0/3040918122 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:41:50.306 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.042+0000 7fa2462b9700 1 -- 172.21.15.105:0/3040918122 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa23c009550 con 0x7fa240105470 2024-04-06T02:41:50.306 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.042+0000 7fa2462b9700 1 --2- 172.21.15.105:0/3040918122 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa240105470 0x7fa240199090 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa23c006af0 tx=0x7fa23c011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:41:50.307 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.042+0000 7fa2377fe700 1 -- 172.21.15.105:0/3040918122 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa23c003b90 con 0x7fa240105470 2024-04-06T02:41:50.307 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.042+0000 7fa2377fe700 1 -- 172.21.15.105:0/3040918122 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa23c01a8a0 con 0x7fa240105470 2024-04-06T02:41:50.307 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.042+0000 7fa24851d700 1 -- 172.21.15.105:0/3040918122 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2401997d0 con 0x7fa240105470 2024-04-06T02:41:50.307 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.042+0000 7fa2377fe700 1 -- 172.21.15.105:0/3040918122 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa23c019880 con 0x7fa240105470 2024-04-06T02:41:50.307 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.042+0000 7fa24851d700 1 -- 172.21.15.105:0/3040918122 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa240199c70 con 0x7fa240105470 2024-04-06T02:41:50.307 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.042+0000 7fa2377fe700 1 -- 172.21.15.105:0/3040918122 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa23c02d440 con 0x7fa240105470 2024-04-06T02:41:50.307 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.042+0000 7fa2377fe700 1 -- 172.21.15.105:0/3040918122 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa23c0248a0 con 0x7fa240105470 2024-04-06T02:41:50.307 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.043+0000 7fa24851d700 1 -- 172.21.15.105:0/3040918122 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2400623c0 con 0x7fa240105470 2024-04-06T02:41:50.307 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.045+0000 7fa2377fe700 1 -- 172.21.15.105:0/3040918122 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fa23c01aa10 con 0x7fa240105470 2024-04-06T02:41:50.307 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.116+0000 7fa24851d700 1 -- 172.21.15.105:0/3040918122 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa240109920 con 0x7fa240105470 2024-04-06T02:41:50.307 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.116+0000 7fa2377fe700 1 -- 172.21.15.105:0/3040918122 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7fa23c01abb0 con 0x7fa240105470 2024-04-06T02:41:50.307 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.118+0000 7fa24851d700 1 -- 172.21.15.105:0/3040918122 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa240105470 msgr2=0x7fa240199090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:41:50.307 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.118+0000 7fa24851d700 1 --2- 172.21.15.105:0/3040918122 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa240105470 0x7fa240199090 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa23c006af0 tx=0x7fa23c011a90 comp rx=0 tx=0).stop 2024-04-06T02:41:50.307 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.118+0000 7fa24851d700 1 -- 172.21.15.105:0/3040918122 shutdown_connections 2024-04-06T02:41:50.308 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.118+0000 7fa24851d700 1 --2- 172.21.15.105:0/3040918122 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa240105470 0x7fa240199090 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:41:50.308 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.118+0000 7fa24851d700 1 -- 172.21.15.105:0/3040918122 >> 172.21.15.105:0/3040918122 conn(0x7fa2401009f0 msgr2=0x7fa240101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:41:50.308 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.118+0000 7fa24851d700 1 -- 172.21.15.105:0/3040918122 shutdown_connections 2024-04-06T02:41:50.308 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:44.118+0000 7fa24851d700 1 -- 172.21.15.105:0/3040918122 wait complete. 2024-04-06T02:41:50.308 INFO:teuthology.orchestra.run.smithi105.stdout:mgr not available, waiting (2/15)... 2024-04-06T02:41:53.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:53 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/1924365036' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-06T02:41:53.567 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-04-06T02:41:53.567 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout { 2024-04-06T02:41:53.567 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "fsid": "130df578-f3bf-11ee-b649-cb9ed24678a4", 2024-04-06T02:41:53.568 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "health": { 2024-04-06T02:41:53.568 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-06T02:41:53.568 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-06T02:41:53.568 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-06T02:41:53.568 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-06T02:41:53.568 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-06T02:41:53.568 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-06T02:41:53.568 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 0 2024-04-06T02:41:53.568 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-04-06T02:41:53.568 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-06T02:41:53.568 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "smithi105" 2024-04-06T02:41:53.568 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-04-06T02:41:53.568 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2024-04-06T02:41:53.568 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-06T02:41:53.569 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T02:41:53.569 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-06T02:41:53.569 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-06T02:41:53.569 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-06T02:41:53.569 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-06T02:41:53.569 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T02:41:53.569 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-06T02:41:53.569 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-06T02:41:53.569 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-06T02:41:53.577 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-06T02:41:53.577 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-06T02:41:53.578 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-06T02:41:53.578 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-06T02:41:53.578 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-06T02:41:53.578 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-06T02:41:53.578 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-06T02:41:53.578 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-06T02:41:53.578 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-06T02:41:53.578 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-06T02:41:53.578 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-06T02:41:53.578 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-06T02:41:53.578 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-06T02:41:53.578 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-06T02:41:53.578 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-06T02:41:53.579 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T02:41:53.579 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-06T02:41:53.579 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-06T02:41:53.579 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-06T02:41:53.579 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-06T02:41:53.579 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-06T02:41:53.579 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-06T02:41:53.579 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-06T02:41:53.579 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-06T02:41:53.579 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-06T02:41:53.579 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "restful" 2024-04-06T02:41:53.580 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-04-06T02:41:53.580 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-06T02:41:53.580 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-06T02:41:53.580 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-06T02:41:53.580 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T02:41:53.580 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "modified": "2024-04-06T02:41:32.645448+0000", 2024-04-06T02:41:53.580 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-06T02:41:53.580 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-06T02:41:53.580 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-06T02:41:53.580 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout } 2024-04-06T02:41:53.580 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.806+0000 7fc33cb87700 1 Processor -- start 2024-04-06T02:41:53.580 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.806+0000 7fc33cb87700 1 -- start start 2024-04-06T02:41:53.580 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.806+0000 7fc33cb87700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc338107020 0x7fc338109410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:41:53.580 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.806+0000 7fc33cb87700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc338079e50 con 0x7fc338107020 2024-04-06T02:41:53.580 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.806+0000 7fc33659c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc338107020 0x7fc338109410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:41:53.581 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.806+0000 7fc33659c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc338107020 0x7fc338109410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:51100/0 (socket says 172.21.15.105:51100) 2024-04-06T02:41:53.581 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.806+0000 7fc33659c700 1 -- 172.21.15.105:0/1474433479 learned_addr learned my addr 172.21.15.105:0/1474433479 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:41:53.581 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.807+0000 7fc33659c700 1 -- 172.21.15.105:0/1474433479 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc338109950 con 0x7fc338107020 2024-04-06T02:41:53.581 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.807+0000 7fc33659c700 1 --2- 172.21.15.105:0/1474433479 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc338107020 0x7fc338109410 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc3280098a0 tx=0x7fc328009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a91e9a400bca112a server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:41:53.581 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.807+0000 7fc33559a700 1 -- 172.21.15.105:0/1474433479 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc328004180 con 0x7fc338107020 2024-04-06T02:41:53.581 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.807+0000 7fc33559a700 1 -- 172.21.15.105:0/1474433479 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc3280042e0 con 0x7fc338107020 2024-04-06T02:41:53.581 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.807+0000 7fc33559a700 1 -- 172.21.15.105:0/1474433479 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc328011a50 con 0x7fc338107020 2024-04-06T02:41:53.581 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.808+0000 7fc33cb87700 1 -- 172.21.15.105:0/1474433479 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc338107020 msgr2=0x7fc338109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:41:53.581 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.808+0000 7fc33cb87700 1 --2- 172.21.15.105:0/1474433479 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc338107020 0x7fc338109410 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc3280098a0 tx=0x7fc328009bb0 comp rx=0 tx=0).stop 2024-04-06T02:41:53.581 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.808+0000 7fc33cb87700 1 -- 172.21.15.105:0/1474433479 shutdown_connections 2024-04-06T02:41:53.581 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.808+0000 7fc33cb87700 1 --2- 172.21.15.105:0/1474433479 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc338107020 0x7fc338109410 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:41:53.581 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.808+0000 7fc33cb87700 1 -- 172.21.15.105:0/1474433479 >> 172.21.15.105:0/1474433479 conn(0x7fc3381009f0 msgr2=0x7fc338102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:41:53.581 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.808+0000 7fc33cb87700 1 -- 172.21.15.105:0/1474433479 shutdown_connections 2024-04-06T02:41:53.581 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.808+0000 7fc33cb87700 1 -- 172.21.15.105:0/1474433479 wait complete. 2024-04-06T02:41:53.582 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.809+0000 7fc33cb87700 1 Processor -- start 2024-04-06T02:41:53.582 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.809+0000 7fc33cb87700 1 -- start start 2024-04-06T02:41:53.582 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.809+0000 7fc33cb87700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc338107020 0x7fc338106980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:41:53.582 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.809+0000 7fc33cb87700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc338106ec0 con 0x7fc338107020 2024-04-06T02:41:53.582 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.809+0000 7fc33659c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc338107020 0x7fc338106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:41:53.582 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.809+0000 7fc33659c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc338107020 0x7fc338106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:51114/0 (socket says 172.21.15.105:51114) 2024-04-06T02:41:53.582 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.809+0000 7fc33659c700 1 -- 172.21.15.105:0/1924365036 learned_addr learned my addr 172.21.15.105:0/1924365036 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:41:53.582 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.809+0000 7fc33659c700 1 -- 172.21.15.105:0/1924365036 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc328009550 con 0x7fc338107020 2024-04-06T02:41:53.582 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.809+0000 7fc33659c700 1 --2- 172.21.15.105:0/1924365036 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc338107020 0x7fc338106980 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc3280119e0 tx=0x7fc328011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:41:53.582 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.809+0000 7fc3277fe700 1 -- 172.21.15.105:0/1924365036 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc328003b90 con 0x7fc338107020 2024-04-06T02:41:53.582 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.810+0000 7fc33cb87700 1 -- 172.21.15.105:0/1924365036 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc338104fd0 con 0x7fc338107020 2024-04-06T02:41:53.582 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.810+0000 7fc3277fe700 1 -- 172.21.15.105:0/1924365036 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc32801a8a0 con 0x7fc338107020 2024-04-06T02:41:53.582 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.810+0000 7fc3277fe700 1 -- 172.21.15.105:0/1924365036 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc328019930 con 0x7fc338107020 2024-04-06T02:41:53.583 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.810+0000 7fc33cb87700 1 -- 172.21.15.105:0/1924365036 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc338105470 con 0x7fc338107020 2024-04-06T02:41:53.583 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.810+0000 7fc3277fe700 1 -- 172.21.15.105:0/1924365036 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 4) v1 ==== 45272+0+0 (secure 0 0 0) 0x7fc328019bf0 con 0x7fc338107020 2024-04-06T02:41:53.583 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.811+0000 7fc3277fe700 1 --2- 172.21.15.105:0/1924365036 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fc320038290 0x7fc32003a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:41:53.583 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.811+0000 7fc3277fe700 1 -- 172.21.15.105:0/1924365036 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc32804d540 con 0x7fc338107020 2024-04-06T02:41:53.583 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.811+0000 7fc335d9b700 1 --2- 172.21.15.105:0/1924365036 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fc320038290 0x7fc32003a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:41:53.583 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.811+0000 7fc33cb87700 1 -- 172.21.15.105:0/1924365036 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc33810a6a0 con 0x7fc338107020 2024-04-06T02:41:53.583 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.811+0000 7fc335d9b700 1 --2- 172.21.15.105:0/1924365036 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fc320038290 0x7fc32003a750 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc32c006fd0 tx=0x7fc32c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:41:53.583 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:52.816+0000 7fc3277fe700 1 -- 172.21.15.105:0/1924365036 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc32802d440 con 0x7fc338107020 2024-04-06T02:41:53.583 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:53.094+0000 7fc33cb87700 1 -- 172.21.15.105:0/1924365036 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fc33810a890 con 0x7fc338107020 2024-04-06T02:41:53.583 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:53.094+0000 7fc3277fe700 1 -- 172.21.15.105:0/1924365036 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7fc32804a020 con 0x7fc338107020 2024-04-06T02:41:53.583 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:53.096+0000 7fc33cb87700 1 -- 172.21.15.105:0/1924365036 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fc320038290 msgr2=0x7fc32003a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:41:53.583 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:53.096+0000 7fc33cb87700 1 --2- 172.21.15.105:0/1924365036 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fc320038290 0x7fc32003a750 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc32c006fd0 tx=0x7fc32c006e40 comp rx=0 tx=0).stop 2024-04-06T02:41:53.583 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:53.096+0000 7fc33cb87700 1 -- 172.21.15.105:0/1924365036 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc338107020 msgr2=0x7fc338106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:41:53.583 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:53.096+0000 7fc33cb87700 1 --2- 172.21.15.105:0/1924365036 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc338107020 0x7fc338106980 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc3280119e0 tx=0x7fc328011ac0 comp rx=0 tx=0).stop 2024-04-06T02:41:53.584 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:53.097+0000 7fc33cb87700 1 -- 172.21.15.105:0/1924365036 shutdown_connections 2024-04-06T02:41:53.584 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:53.097+0000 7fc33cb87700 1 --2- 172.21.15.105:0/1924365036 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fc320038290 0x7fc32003a750 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:41:53.584 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:53.097+0000 7fc33cb87700 1 --2- 172.21.15.105:0/1924365036 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc338107020 0x7fc338106980 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:41:53.584 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:53.097+0000 7fc33cb87700 1 -- 172.21.15.105:0/1924365036 >> 172.21.15.105:0/1924365036 conn(0x7fc3381009f0 msgr2=0x7fc338101720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:41:53.584 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:53.097+0000 7fc33cb87700 1 -- 172.21.15.105:0/1924365036 shutdown_connections 2024-04-06T02:41:53.584 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:53.097+0000 7fc33cb87700 1 -- 172.21.15.105:0/1924365036 wait complete. 2024-04-06T02:41:53.584 INFO:teuthology.orchestra.run.smithi105.stdout:mgr is available 2024-04-06T02:41:54.510 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:54 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/3386778484' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-06T02:41:54.778 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-04-06T02:41:54.778 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout [global] 2024-04-06T02:41:54.778 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout fsid = 130df578-f3bf-11ee-b649-cb9ed24678a4 2024-04-06T02:41:54.778 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-06T02:41:54.778 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-06T02:41:54.779 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-06T02:41:54.779 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-06T02:41:54.779 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-04-06T02:41:54.779 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-06T02:41:54.779 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-06T02:41:54.779 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-04-06T02:41:54.779 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout [osd] 2024-04-06T02:41:54.779 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-06T02:41:54.779 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-06T02:41:54.779 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-06T02:41:54.779 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-06T02:41:54.779 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-06T02:41:54.779 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.021+0000 7fae065ce700 1 Processor -- start 2024-04-06T02:41:54.779 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.021+0000 7fae065ce700 1 -- start start 2024-04-06T02:41:54.779 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.021+0000 7fae065ce700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae00104e10 0x7fae00105230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:41:54.779 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.021+0000 7fae065ce700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae00074640 con 0x7fae00104e10 2024-04-06T02:41:54.780 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.022+0000 7fadfffff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae00104e10 0x7fae00105230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:41:54.780 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.022+0000 7fadfffff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae00104e10 0x7fae00105230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:51116/0 (socket says 172.21.15.105:51116) 2024-04-06T02:41:54.780 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.022+0000 7fadfffff700 1 -- 172.21.15.105:0/3365254793 learned_addr learned my addr 172.21.15.105:0/3365254793 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:41:54.780 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.022+0000 7fadfffff700 1 -- 172.21.15.105:0/3365254793 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae00105770 con 0x7fae00104e10 2024-04-06T02:41:54.780 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.022+0000 7fadfffff700 1 --2- 172.21.15.105:0/3365254793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae00104e10 0x7fae00105230 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fadf00098a0 tx=0x7fadf0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e45c6e854f5cf98 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:41:54.780 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.022+0000 7fadfeffd700 1 -- 172.21.15.105:0/3365254793 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadf0004180 con 0x7fae00104e10 2024-04-06T02:41:54.780 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.022+0000 7fadfeffd700 1 -- 172.21.15.105:0/3365254793 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fadf00042e0 con 0x7fae00104e10 2024-04-06T02:41:54.780 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.022+0000 7fadfeffd700 1 -- 172.21.15.105:0/3365254793 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadf0011a50 con 0x7fae00104e10 2024-04-06T02:41:54.780 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.023+0000 7fae065ce700 1 -- 172.21.15.105:0/3365254793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae00104e10 msgr2=0x7fae00105230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:41:54.780 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.023+0000 7fae065ce700 1 --2- 172.21.15.105:0/3365254793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae00104e10 0x7fae00105230 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fadf00098a0 tx=0x7fadf0009bb0 comp rx=0 tx=0).stop 2024-04-06T02:41:54.780 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.023+0000 7fae065ce700 1 -- 172.21.15.105:0/3365254793 shutdown_connections 2024-04-06T02:41:54.780 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.023+0000 7fae065ce700 1 --2- 172.21.15.105:0/3365254793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae00104e10 0x7fae00105230 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:41:54.780 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.023+0000 7fae065ce700 1 -- 172.21.15.105:0/3365254793 >> 172.21.15.105:0/3365254793 conn(0x7fae001009f0 msgr2=0x7fae00102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:41:54.780 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.023+0000 7fae065ce700 1 -- 172.21.15.105:0/3365254793 shutdown_connections 2024-04-06T02:41:54.780 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.023+0000 7fae065ce700 1 -- 172.21.15.105:0/3365254793 wait complete. 2024-04-06T02:41:54.781 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.023+0000 7fae065ce700 1 Processor -- start 2024-04-06T02:41:54.781 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.023+0000 7fae065ce700 1 -- start start 2024-04-06T02:41:54.781 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.024+0000 7fae065ce700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae00104e10 0x7fae0019c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:41:54.781 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.024+0000 7fae065ce700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0019c8c0 con 0x7fae00104e10 2024-04-06T02:41:54.781 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.024+0000 7fadfffff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae00104e10 0x7fae0019c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:41:54.781 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.024+0000 7fadfffff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae00104e10 0x7fae0019c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:51122/0 (socket says 172.21.15.105:51122) 2024-04-06T02:41:54.781 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.024+0000 7fadfffff700 1 -- 172.21.15.105:0/3386778484 learned_addr learned my addr 172.21.15.105:0/3386778484 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:41:54.781 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.024+0000 7fadfffff700 1 -- 172.21.15.105:0/3386778484 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf0009550 con 0x7fae00104e10 2024-04-06T02:41:54.781 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.024+0000 7fadfffff700 1 --2- 172.21.15.105:0/3386778484 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae00104e10 0x7fae0019c380 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fadf0000c00 tx=0x7fadf0011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:41:54.781 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.024+0000 7fadfd7fa700 1 -- 172.21.15.105:0/3386778484 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadf0003b90 con 0x7fae00104e10 2024-04-06T02:41:54.781 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.024+0000 7fadfd7fa700 1 -- 172.21.15.105:0/3386778484 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fadf00248b0 con 0x7fae00104e10 2024-04-06T02:41:54.781 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.024+0000 7fae065ce700 1 -- 172.21.15.105:0/3386778484 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae0019cac0 con 0x7fae00104e10 2024-04-06T02:41:54.781 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.024+0000 7fadfd7fa700 1 -- 172.21.15.105:0/3386778484 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadf001a8e0 con 0x7fae00104e10 2024-04-06T02:41:54.781 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.024+0000 7fae065ce700 1 -- 172.21.15.105:0/3386778484 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae0019cf60 con 0x7fae00104e10 2024-04-06T02:41:54.782 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.025+0000 7fadfd7fa700 1 -- 172.21.15.105:0/3386778484 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 4) v1 ==== 45272+0+0 (secure 0 0 0) 0x7fadf001ab50 con 0x7fae00104e10 2024-04-06T02:41:54.782 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.025+0000 7fadfd7fa700 1 --2- 172.21.15.105:0/3386778484 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fade8038320 0x7fade803a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:41:54.782 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.025+0000 7fae065ce700 1 -- 172.21.15.105:0/3386778484 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae00195c30 con 0x7fae00104e10 2024-04-06T02:41:54.782 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.025+0000 7fadfd7fa700 1 -- 172.21.15.105:0/3386778484 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fadf004bff0 con 0x7fae00104e10 2024-04-06T02:41:54.782 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.025+0000 7fadff7fe700 1 --2- 172.21.15.105:0/3386778484 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fade8038320 0x7fade803a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:41:54.782 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.026+0000 7fadff7fe700 1 --2- 172.21.15.105:0/3386778484 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fade8038320 0x7fade803a7e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fadf4006fd0 tx=0x7fadf4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:41:54.782 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.030+0000 7fadfd7fa700 1 -- 172.21.15.105:0/3386778484 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fadf001ae00 con 0x7fae00104e10 2024-04-06T02:41:54.782 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.218+0000 7fae065ce700 1 -- 172.21.15.105:0/3386778484 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fae0002c2c0 con 0x7fae00104e10 2024-04-06T02:41:54.782 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.219+0000 7fadfd7fa700 1 -- 172.21.15.105:0/3386778484 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fadf0049020 con 0x7fae00104e10 2024-04-06T02:41:54.782 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.221+0000 7fae065ce700 1 -- 172.21.15.105:0/3386778484 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fade8038320 msgr2=0x7fade803a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:41:54.782 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.221+0000 7fae065ce700 1 --2- 172.21.15.105:0/3386778484 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fade8038320 0x7fade803a7e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fadf4006fd0 tx=0x7fadf4006e40 comp rx=0 tx=0).stop 2024-04-06T02:41:54.782 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.221+0000 7fae065ce700 1 -- 172.21.15.105:0/3386778484 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae00104e10 msgr2=0x7fae0019c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:41:54.782 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.221+0000 7fae065ce700 1 --2- 172.21.15.105:0/3386778484 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae00104e10 0x7fae0019c380 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fadf0000c00 tx=0x7fadf0011ab0 comp rx=0 tx=0).stop 2024-04-06T02:41:54.783 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.221+0000 7fae065ce700 1 -- 172.21.15.105:0/3386778484 shutdown_connections 2024-04-06T02:41:54.783 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.221+0000 7fae065ce700 1 --2- 172.21.15.105:0/3386778484 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fade8038320 0x7fade803a7e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:41:54.783 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.221+0000 7fae065ce700 1 --2- 172.21.15.105:0/3386778484 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae00104e10 0x7fae0019c380 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:41:54.783 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.221+0000 7fae065ce700 1 -- 172.21.15.105:0/3386778484 >> 172.21.15.105:0/3386778484 conn(0x7fae001009f0 msgr2=0x7fae001909d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:41:54.783 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.221+0000 7fae065ce700 1 -- 172.21.15.105:0/3386778484 shutdown_connections 2024-04-06T02:41:54.783 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:54.221+0000 7fae065ce700 1 -- 172.21.15.105:0/3386778484 wait complete. 2024-04-06T02:41:54.783 INFO:teuthology.orchestra.run.smithi105.stdout:Enabling cephadm module... 2024-04-06T02:41:55.673 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:55 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/3368940772' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-06T02:41:56.501 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.169+0000 7f7e81891700 1 Processor -- start 2024-04-06T02:41:56.502 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.169+0000 7f7e81891700 1 -- start start 2024-04-06T02:41:56.502 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.169+0000 7f7e81891700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e7c07be80 0x7f7e7c07a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:41:56.502 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.169+0000 7f7e81891700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c07a8b0 con 0x7f7e7c07be80 2024-04-06T02:41:56.502 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.169+0000 7f7e7affd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e7c07be80 0x7f7e7c07a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:41:56.502 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.169+0000 7f7e7affd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e7c07be80 0x7f7e7c07a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:51128/0 (socket says 172.21.15.105:51128) 2024-04-06T02:41:56.502 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.169+0000 7f7e7affd700 1 -- 172.21.15.105:0/811311730 learned_addr learned my addr 172.21.15.105:0/811311730 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:41:56.502 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.169+0000 7f7e7affd700 1 -- 172.21.15.105:0/811311730 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7c07a9f0 con 0x7f7e7c07be80 2024-04-06T02:41:56.502 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.170+0000 7f7e7affd700 1 --2- 172.21.15.105:0/811311730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e7c07be80 0x7f7e7c07a2e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7e74009e80 tx=0x7f7e7400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5a13d1902e23bb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:41:56.502 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.170+0000 7f7e79ffb700 1 -- 172.21.15.105:0/811311730 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e7400e070 con 0x7f7e7c07be80 2024-04-06T02:41:56.502 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.170+0000 7f7e79ffb700 1 -- 172.21.15.105:0/811311730 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7e74003e90 con 0x7f7e7c07be80 2024-04-06T02:41:56.503 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.170+0000 7f7e79ffb700 1 -- 172.21.15.105:0/811311730 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e74004380 con 0x7f7e7c07be80 2024-04-06T02:41:56.503 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.170+0000 7f7e81891700 1 -- 172.21.15.105:0/811311730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e7c07be80 msgr2=0x7f7e7c07a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:41:56.503 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.170+0000 7f7e81891700 1 --2- 172.21.15.105:0/811311730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e7c07be80 0x7f7e7c07a2e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7e74009e80 tx=0x7f7e7400b330 comp rx=0 tx=0).stop 2024-04-06T02:41:56.503 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.170+0000 7f7e81891700 1 -- 172.21.15.105:0/811311730 shutdown_connections 2024-04-06T02:41:56.503 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.170+0000 7f7e81891700 1 --2- 172.21.15.105:0/811311730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e7c07be80 0x7f7e7c07a2e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:41:56.503 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.170+0000 7f7e81891700 1 -- 172.21.15.105:0/811311730 >> 172.21.15.105:0/811311730 conn(0x7f7e7c101b40 msgr2=0x7f7e7c103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:41:56.503 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.170+0000 7f7e81891700 1 -- 172.21.15.105:0/811311730 shutdown_connections 2024-04-06T02:41:56.503 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.170+0000 7f7e81891700 1 -- 172.21.15.105:0/811311730 wait complete. 2024-04-06T02:41:56.503 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.171+0000 7f7e81891700 1 Processor -- start 2024-04-06T02:41:56.503 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.171+0000 7f7e81891700 1 -- start start 2024-04-06T02:41:56.503 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.171+0000 7f7e81891700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e7c07be80 0x7f7e7c19c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:41:56.503 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.171+0000 7f7e81891700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c19c920 con 0x7f7e7c07be80 2024-04-06T02:41:56.503 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.171+0000 7f7e7affd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e7c07be80 0x7f7e7c19c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:41:56.503 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.171+0000 7f7e7affd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e7c07be80 0x7f7e7c19c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:51140/0 (socket says 172.21.15.105:51140) 2024-04-06T02:41:56.503 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.171+0000 7f7e7affd700 1 -- 172.21.15.105:0/3368940772 learned_addr learned my addr 172.21.15.105:0/3368940772 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:41:56.503 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.172+0000 7f7e7affd700 1 -- 172.21.15.105:0/3368940772 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e74009b30 con 0x7f7e7c07be80 2024-04-06T02:41:56.504 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.172+0000 7f7e7affd700 1 --2- 172.21.15.105:0/3368940772 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e7c07be80 0x7f7e7c19c3e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7e7400d010 tx=0x7f7e74004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:41:56.504 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.172+0000 7f7e8088f700 1 -- 172.21.15.105:0/3368940772 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e7400e040 con 0x7f7e7c07be80 2024-04-06T02:41:56.504 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.172+0000 7f7e81891700 1 -- 172.21.15.105:0/3368940772 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e7c19cb20 con 0x7f7e7c07be80 2024-04-06T02:41:56.504 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.172+0000 7f7e8088f700 1 -- 172.21.15.105:0/3368940772 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7e74003c30 con 0x7f7e7c07be80 2024-04-06T02:41:56.504 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.172+0000 7f7e8088f700 1 -- 172.21.15.105:0/3368940772 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e7401b790 con 0x7f7e7c07be80 2024-04-06T02:41:56.504 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.172+0000 7f7e81891700 1 -- 172.21.15.105:0/3368940772 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e7c19d020 con 0x7f7e7c07be80 2024-04-06T02:41:56.504 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.173+0000 7f7e8088f700 1 -- 172.21.15.105:0/3368940772 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 4) v1 ==== 45272+0+0 (secure 0 0 0) 0x7f7e74019040 con 0x7f7e7c07be80 2024-04-06T02:41:56.504 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.173+0000 7f7e81891700 1 -- 172.21.15.105:0/3368940772 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e68005320 con 0x7f7e7c07be80 2024-04-06T02:41:56.504 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.173+0000 7f7e8088f700 1 --2- 172.21.15.105:0/3368940772 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7f7e64038290 0x7f7e6403a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:41:56.504 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.173+0000 7f7e8088f700 1 -- 172.21.15.105:0/3368940772 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7e7404bdc0 con 0x7f7e7c07be80 2024-04-06T02:41:56.504 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.173+0000 7f7e7a7fc700 1 --2- 172.21.15.105:0/3368940772 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7f7e64038290 0x7f7e6403a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:41:56.504 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.173+0000 7f7e7a7fc700 1 --2- 172.21.15.105:0/3368940772 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7f7e64038290 0x7f7e6403a750 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7e70006fd0 tx=0x7f7e70006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:41:56.504 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.178+0000 7f7e8088f700 1 -- 172.21.15.105:0/3368940772 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7e74011980 con 0x7f7e7c07be80 2024-04-06T02:41:56.504 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.414+0000 7f7e81891700 1 -- 172.21.15.105:0/3368940772 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f7e680059f0 con 0x7f7e7c07be80 2024-04-06T02:41:56.505 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.466+0000 7f7e8088f700 1 -- 172.21.15.105:0/3368940772 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 5) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f7e7401a390 con 0x7f7e7c07be80 2024-04-06T02:41:56.505 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.468+0000 7f7e8088f700 1 -- 172.21.15.105:0/3368940772 <== mon.0 v2:172.21.15.105:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f7e74021df0 con 0x7f7e7c07be80 2024-04-06T02:41:56.505 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.471+0000 7f7e81891700 1 -- 172.21.15.105:0/3368940772 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7f7e64038290 msgr2=0x7f7e6403a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:41:56.505 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.471+0000 7f7e81891700 1 --2- 172.21.15.105:0/3368940772 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7f7e64038290 0x7f7e6403a750 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7e70006fd0 tx=0x7f7e70006e40 comp rx=0 tx=0).stop 2024-04-06T02:41:56.505 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.471+0000 7f7e81891700 1 -- 172.21.15.105:0/3368940772 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e7c07be80 msgr2=0x7f7e7c19c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:41:56.506 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.471+0000 7f7e81891700 1 --2- 172.21.15.105:0/3368940772 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e7c07be80 0x7f7e7c19c3e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7e7400d010 tx=0x7f7e74004430 comp rx=0 tx=0).stop 2024-04-06T02:41:56.506 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.471+0000 7f7e81891700 1 -- 172.21.15.105:0/3368940772 shutdown_connections 2024-04-06T02:41:56.506 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.471+0000 7f7e81891700 1 --2- 172.21.15.105:0/3368940772 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7f7e64038290 0x7f7e6403a750 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:41:56.506 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.471+0000 7f7e81891700 1 --2- 172.21.15.105:0/3368940772 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e7c07be80 0x7f7e7c19c3e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:41:56.506 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.471+0000 7f7e81891700 1 -- 172.21.15.105:0/3368940772 >> 172.21.15.105:0/3368940772 conn(0x7f7e7c101b40 msgr2=0x7f7e7c1028c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:41:56.507 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.471+0000 7f7e81891700 1 -- 172.21.15.105:0/3368940772 shutdown_connections 2024-04-06T02:41:56.507 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:55.471+0000 7f7e81891700 1 -- 172.21.15.105:0/3368940772 wait complete. 2024-04-06T02:41:56.723 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:56 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/3368940772' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-06T02:41:56.724 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:56 smithi105 ceph-mon[29228]: mgrmap e5: smithi105.vqqcpx(active, since 9s) 2024-04-06T02:41:57.673 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:41:57 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/4221473856' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-06T02:41:57.808 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout { 2024-04-06T02:41:57.808 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-06T02:41:57.808 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-06T02:41:57.808 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "active_name": "smithi105.vqqcpx", 2024-04-06T02:41:57.808 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-06T02:41:57.808 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout } 2024-04-06T02:41:57.808 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.945+0000 7fec62288700 1 Processor -- start 2024-04-06T02:41:57.808 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.945+0000 7fec62288700 1 -- start start 2024-04-06T02:41:57.808 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.945+0000 7fec62288700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec5c106560 0x7fec5c106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:41:57.809 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.945+0000 7fec62288700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c106f50 con 0x7fec5c106560 2024-04-06T02:41:57.809 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.946+0000 7fec5bfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec5c106560 0x7fec5c106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:41:57.809 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.946+0000 7fec5bfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec5c106560 0x7fec5c106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:51160/0 (socket says 172.21.15.105:51160) 2024-04-06T02:41:57.809 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.946+0000 7fec5bfff700 1 -- 172.21.15.105:0/2950314397 learned_addr learned my addr 172.21.15.105:0/2950314397 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:41:57.809 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.946+0000 7fec5bfff700 1 -- 172.21.15.105:0/2950314397 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec5c107760 con 0x7fec5c106560 2024-04-06T02:41:57.809 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.946+0000 7fec5bfff700 1 --2- 172.21.15.105:0/2950314397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec5c106560 0x7fec5c106980 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fec440098a0 tx=0x7fec44009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d14346c3eac12f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:41:57.809 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.946+0000 7fec5affd700 1 -- 172.21.15.105:0/2950314397 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec44004180 con 0x7fec5c106560 2024-04-06T02:41:57.809 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.946+0000 7fec5affd700 1 -- 172.21.15.105:0/2950314397 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fec440042e0 con 0x7fec5c106560 2024-04-06T02:41:57.809 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.946+0000 7fec5affd700 1 -- 172.21.15.105:0/2950314397 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec44011a50 con 0x7fec5c106560 2024-04-06T02:41:57.809 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.947+0000 7fec62288700 1 -- 172.21.15.105:0/2950314397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec5c106560 msgr2=0x7fec5c106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:41:57.810 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.947+0000 7fec62288700 1 --2- 172.21.15.105:0/2950314397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec5c106560 0x7fec5c106980 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fec440098a0 tx=0x7fec44009bb0 comp rx=0 tx=0).stop 2024-04-06T02:41:57.810 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.947+0000 7fec62288700 1 -- 172.21.15.105:0/2950314397 shutdown_connections 2024-04-06T02:41:57.810 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.947+0000 7fec62288700 1 --2- 172.21.15.105:0/2950314397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec5c106560 0x7fec5c106980 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:41:57.810 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.947+0000 7fec62288700 1 -- 172.21.15.105:0/2950314397 >> 172.21.15.105:0/2950314397 conn(0x7fec5c101b40 msgr2=0x7fec5c103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:41:57.810 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.947+0000 7fec62288700 1 -- 172.21.15.105:0/2950314397 shutdown_connections 2024-04-06T02:41:57.810 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.947+0000 7fec62288700 1 -- 172.21.15.105:0/2950314397 wait complete. 2024-04-06T02:41:57.810 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.948+0000 7fec62288700 1 Processor -- start 2024-04-06T02:41:57.810 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.948+0000 7fec62288700 1 -- start start 2024-04-06T02:41:57.810 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.948+0000 7fec62288700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec5c106560 0x7fec5c19a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:41:57.810 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.948+0000 7fec62288700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c19a700 con 0x7fec5c106560 2024-04-06T02:41:57.810 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.948+0000 7fec5bfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec5c106560 0x7fec5c19a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:41:57.810 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.948+0000 7fec5bfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec5c106560 0x7fec5c19a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:51170/0 (socket says 172.21.15.105:51170) 2024-04-06T02:41:57.810 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.948+0000 7fec5bfff700 1 -- 172.21.15.105:0/4221473856 learned_addr learned my addr 172.21.15.105:0/4221473856 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:41:57.810 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.949+0000 7fec5bfff700 1 -- 172.21.15.105:0/4221473856 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec44009550 con 0x7fec5c106560 2024-04-06T02:41:57.811 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.949+0000 7fec5bfff700 1 --2- 172.21.15.105:0/4221473856 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec5c106560 0x7fec5c19a1c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fec44000c00 tx=0x7fec44011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:41:57.811 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.949+0000 7fec597fa700 1 -- 172.21.15.105:0/4221473856 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec44003b90 con 0x7fec5c106560 2024-04-06T02:41:57.811 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.949+0000 7fec597fa700 1 -- 172.21.15.105:0/4221473856 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fec440248b0 con 0x7fec5c106560 2024-04-06T02:41:57.811 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.949+0000 7fec62288700 1 -- 172.21.15.105:0/4221473856 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec5c19a900 con 0x7fec5c106560 2024-04-06T02:41:57.811 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.949+0000 7fec597fa700 1 -- 172.21.15.105:0/4221473856 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec4401a8e0 con 0x7fec5c106560 2024-04-06T02:41:57.811 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.949+0000 7fec62288700 1 -- 172.21.15.105:0/4221473856 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec5c19ada0 con 0x7fec5c106560 2024-04-06T02:41:57.811 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.950+0000 7fec597fa700 1 -- 172.21.15.105:0/4221473856 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 5) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fec4401aba0 con 0x7fec5c106560 2024-04-06T02:41:57.811 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.950+0000 7fec62288700 1 -- 172.21.15.105:0/4221473856 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec5c0623c0 con 0x7fec5c106560 2024-04-06T02:41:57.811 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.950+0000 7fec597fa700 1 --2- 172.21.15.105:0/4221473856 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fec48038320 0x7fec4803a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:41:57.811 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.950+0000 7fec597fa700 1 -- 172.21.15.105:0/4221473856 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fec4404c0b0 con 0x7fec5c106560 2024-04-06T02:41:57.811 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.950+0000 7fec5b7fe700 1 -- 172.21.15.105:0/4221473856 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fec48038320 msgr2=0x7fec4803a7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/3245193613 2024-04-06T02:41:57.811 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.950+0000 7fec5b7fe700 1 --2- 172.21.15.105:0/4221473856 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fec48038320 0x7fec4803a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T02:41:57.811 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:56.955+0000 7fec597fa700 1 -- 172.21.15.105:0/4221473856 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fec440233a0 con 0x7fec5c106560 2024-04-06T02:41:57.812 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:57.151+0000 7fec5b7fe700 1 -- 172.21.15.105:0/4221473856 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fec48038320 msgr2=0x7fec4803a7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/3245193613 2024-04-06T02:41:57.812 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:57.151+0000 7fec5b7fe700 1 --2- 172.21.15.105:0/4221473856 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fec48038320 0x7fec4803a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T02:41:57.812 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:57.207+0000 7fec62288700 1 -- 172.21.15.105:0/4221473856 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fec5c19db80 con 0x7fec5c106560 2024-04-06T02:41:57.812 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:57.207+0000 7fec597fa700 1 -- 172.21.15.105:0/4221473856 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fec4402c350 con 0x7fec5c106560 2024-04-06T02:41:57.812 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:57.209+0000 7fec62288700 1 -- 172.21.15.105:0/4221473856 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fec48038320 msgr2=0x7fec4803a7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T02:41:57.812 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:57.209+0000 7fec62288700 1 --2- 172.21.15.105:0/4221473856 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fec48038320 0x7fec4803a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:41:57.812 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:57.209+0000 7fec62288700 1 -- 172.21.15.105:0/4221473856 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec5c106560 msgr2=0x7fec5c19a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:41:57.812 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:57.209+0000 7fec62288700 1 --2- 172.21.15.105:0/4221473856 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec5c106560 0x7fec5c19a1c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fec44000c00 tx=0x7fec44011ab0 comp rx=0 tx=0).stop 2024-04-06T02:41:57.812 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:57.209+0000 7fec62288700 1 -- 172.21.15.105:0/4221473856 shutdown_connections 2024-04-06T02:41:57.812 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:57.209+0000 7fec62288700 1 --2- 172.21.15.105:0/4221473856 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fec48038320 0x7fec4803a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:41:57.812 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:57.209+0000 7fec62288700 1 --2- 172.21.15.105:0/4221473856 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fec5c106560 0x7fec5c19a1c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:41:57.812 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:57.209+0000 7fec62288700 1 -- 172.21.15.105:0/4221473856 >> 172.21.15.105:0/4221473856 conn(0x7fec5c101b40 msgr2=0x7fec5c102870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:41:57.812 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:57.209+0000 7fec62288700 1 -- 172.21.15.105:0/4221473856 shutdown_connections 2024-04-06T02:41:57.812 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:57.209+0000 7fec62288700 1 -- 172.21.15.105:0/4221473856 wait complete. 2024-04-06T02:41:57.812 INFO:teuthology.orchestra.run.smithi105.stdout:Waiting for the mgr to restart... 2024-04-06T02:41:57.813 INFO:teuthology.orchestra.run.smithi105.stdout:Waiting for mgr epoch 5... 2024-04-06T02:42:02.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:02 smithi105 ceph-mon[29228]: Active manager daemon smithi105.vqqcpx restarted 2024-04-06T02:42:02.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:02 smithi105 ceph-mon[29228]: Activating manager daemon smithi105.vqqcpx 2024-04-06T02:42:02.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:02 smithi105 ceph-mon[29228]: osdmap e2: 0 total, 0 up, 0 in 2024-04-06T02:42:02.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:02 smithi105 ceph-mon[29228]: mgrmap e6: smithi105.vqqcpx(active, starting, since 0.0032522s) 2024-04-06T02:42:02.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:02 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-04-06T02:42:02.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:02 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mgr metadata", "who": "smithi105.vqqcpx", "id": "smithi105.vqqcpx"}]: dispatch 2024-04-06T02:42:02.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:02 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T02:42:02.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:02 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T02:42:02.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:02 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T02:42:02.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:02 smithi105 ceph-mon[29228]: Manager daemon smithi105.vqqcpx is now available 2024-04-06T02:42:02.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:02 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:02.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:02 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:02.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:02 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:42:02.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:02 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:42:02.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:02 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:42:03.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:03 smithi105 ceph-mon[29228]: Found migration_current of "None". Setting to last migration. 2024-04-06T02:42:03.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:03 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi105.vqqcpx/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:42:03.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:03 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi105.vqqcpx/trash_purge_schedule"}]: dispatch 2024-04-06T02:42:03.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:03 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:03.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:03 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:03.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:03 smithi105 ceph-mon[29228]: mgrmap e7: smithi105.vqqcpx(active, since 1.00546s) 2024-04-06T02:42:03.774 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout { 2024-04-06T02:42:03.774 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-06T02:42:03.774 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-06T02:42:03.774 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout } 2024-04-06T02:42:03.774 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.218+0000 7fcab4ca4700 1 Processor -- start 2024-04-06T02:42:03.774 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.218+0000 7fcab4ca4700 1 -- start start 2024-04-06T02:42:03.774 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.218+0000 7fcab4ca4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcab0107040 0x7fcab0109430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:03.775 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.218+0000 7fcab4ca4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab0079e30 con 0x7fcab0107040 2024-04-06T02:42:03.775 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.218+0000 7fcaae59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcab0107040 0x7fcab0109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:03.775 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.219+0000 7fcaae59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcab0107040 0x7fcab0109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:51184/0 (socket says 172.21.15.105:51184) 2024-04-06T02:42:03.775 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.219+0000 7fcaae59c700 1 -- 172.21.15.105:0/3580616125 learned_addr learned my addr 172.21.15.105:0/3580616125 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:03.775 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.219+0000 7fcaae59c700 1 -- 172.21.15.105:0/3580616125 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab0109970 con 0x7fcab0107040 2024-04-06T02:42:03.775 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.219+0000 7fcaae59c700 1 --2- 172.21.15.105:0/3580616125 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcab0107040 0x7fcab0109430 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcaa40098a0 tx=0x7fcaa4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e280643c2c6b67d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:03.775 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.219+0000 7fcaad59a700 1 -- 172.21.15.105:0/3580616125 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcaa4004180 con 0x7fcab0107040 2024-04-06T02:42:03.775 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.219+0000 7fcaad59a700 1 -- 172.21.15.105:0/3580616125 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcaa40042e0 con 0x7fcab0107040 2024-04-06T02:42:03.775 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.219+0000 7fcaad59a700 1 -- 172.21.15.105:0/3580616125 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcaa4011a50 con 0x7fcab0107040 2024-04-06T02:42:03.775 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.220+0000 7fcab4ca4700 1 -- 172.21.15.105:0/3580616125 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcab0107040 msgr2=0x7fcab0109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:03.775 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.220+0000 7fcab4ca4700 1 --2- 172.21.15.105:0/3580616125 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcab0107040 0x7fcab0109430 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcaa40098a0 tx=0x7fcaa4009bb0 comp rx=0 tx=0).stop 2024-04-06T02:42:03.775 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.220+0000 7fcab4ca4700 1 -- 172.21.15.105:0/3580616125 shutdown_connections 2024-04-06T02:42:03.775 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.220+0000 7fcab4ca4700 1 --2- 172.21.15.105:0/3580616125 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcab0107040 0x7fcab0109430 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:03.775 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.220+0000 7fcab4ca4700 1 -- 172.21.15.105:0/3580616125 >> 172.21.15.105:0/3580616125 conn(0x7fcab01009f0 msgr2=0x7fcab0102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:03.775 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.220+0000 7fcab4ca4700 1 -- 172.21.15.105:0/3580616125 shutdown_connections 2024-04-06T02:42:03.776 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.220+0000 7fcab4ca4700 1 -- 172.21.15.105:0/3580616125 wait complete. 2024-04-06T02:42:03.776 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.221+0000 7fcab4ca4700 1 Processor -- start 2024-04-06T02:42:03.776 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.221+0000 7fcab4ca4700 1 -- start start 2024-04-06T02:42:03.776 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.221+0000 7fcab4ca4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcab0107040 0x7fcab019a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:03.776 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.221+0000 7fcab4ca4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab019a6a0 con 0x7fcab0107040 2024-04-06T02:42:03.776 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.221+0000 7fcaae59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcab0107040 0x7fcab019a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:03.776 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.221+0000 7fcaae59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcab0107040 0x7fcab019a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:51190/0 (socket says 172.21.15.105:51190) 2024-04-06T02:42:03.776 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.221+0000 7fcaae59c700 1 -- 172.21.15.105:0/3693870526 learned_addr learned my addr 172.21.15.105:0/3693870526 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:03.776 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.221+0000 7fcaae59c700 1 -- 172.21.15.105:0/3693870526 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa4009550 con 0x7fcab0107040 2024-04-06T02:42:03.776 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.221+0000 7fcaae59c700 1 --2- 172.21.15.105:0/3693870526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcab0107040 0x7fcab019a160 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcaa40119e0 tx=0x7fcaa4011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:03.776 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.221+0000 7fca9f7fe700 1 -- 172.21.15.105:0/3693870526 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcaa4003b90 con 0x7fcab0107040 2024-04-06T02:42:03.776 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.222+0000 7fca9f7fe700 1 -- 172.21.15.105:0/3693870526 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcaa401a8a0 con 0x7fcab0107040 2024-04-06T02:42:03.776 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.222+0000 7fca9f7fe700 1 -- 172.21.15.105:0/3693870526 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcaa4019930 con 0x7fcab0107040 2024-04-06T02:42:03.776 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.222+0000 7fcab4ca4700 1 -- 172.21.15.105:0/3693870526 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcab019a8a0 con 0x7fcab0107040 2024-04-06T02:42:03.777 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.222+0000 7fcab4ca4700 1 -- 172.21.15.105:0/3693870526 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcab019ad40 con 0x7fcab0107040 2024-04-06T02:42:03.777 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.222+0000 7fca9f7fe700 1 -- 172.21.15.105:0/3693870526 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 5) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fcaa4019ba0 con 0x7fcab0107040 2024-04-06T02:42:03.777 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.223+0000 7fca9f7fe700 1 --2- 172.21.15.105:0/3693870526 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fca98038330 0x7fca9803a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:03.777 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.223+0000 7fcab4ca4700 1 -- 172.21.15.105:0/3693870526 --> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcab010a690 con 0x7fca98038330 2024-04-06T02:42:03.777 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.223+0000 7fca9f7fe700 1 -- 172.21.15.105:0/3693870526 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fcaa404c6a0 con 0x7fcab0107040 2024-04-06T02:42:03.777 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.223+0000 7fcaadd9b700 1 -- 172.21.15.105:0/3693870526 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fca98038330 msgr2=0x7fca9803a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/3245193613 2024-04-06T02:42:03.777 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.223+0000 7fcaadd9b700 1 --2- 172.21.15.105:0/3693870526 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fca98038330 0x7fca9803a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T02:42:03.777 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.424+0000 7fcaadd9b700 1 -- 172.21.15.105:0/3693870526 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fca98038330 msgr2=0x7fca9803a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/3245193613 2024-04-06T02:42:03.777 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.424+0000 7fcaadd9b700 1 --2- 172.21.15.105:0/3693870526 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fca98038330 0x7fca9803a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T02:42:03.777 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.825+0000 7fcaadd9b700 1 -- 172.21.15.105:0/3693870526 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fca98038330 msgr2=0x7fca9803a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/3245193613 2024-04-06T02:42:03.777 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:58.825+0000 7fcaadd9b700 1 --2- 172.21.15.105:0/3693870526 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fca98038330 0x7fca9803a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-06T02:42:03.777 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:59.626+0000 7fcaadd9b700 1 -- 172.21.15.105:0/3693870526 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fca98038330 msgr2=0x7fca9803a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/3245193613 2024-04-06T02:42:03.777 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:41:59.626+0000 7fcaadd9b700 1 --2- 172.21.15.105:0/3693870526 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fca98038330 0x7fca9803a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-06T02:42:03.778 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:01.227+0000 7fcaadd9b700 1 -- 172.21.15.105:0/3693870526 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fca98038330 msgr2=0x7fca9803a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/3245193613 2024-04-06T02:42:03.778 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:01.227+0000 7fcaadd9b700 1 --2- 172.21.15.105:0/3693870526 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fca98038330 0x7fca9803a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-06T02:42:03.778 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:02.155+0000 7fca9f7fe700 1 -- 172.21.15.105:0/3693870526 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mgrmap(e 6) v1 ==== 45050+0+0 (secure 0 0 0) 0x7fcaa402c3a0 con 0x7fcab0107040 2024-04-06T02:42:03.778 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:02.156+0000 7fca9f7fe700 1 -- 172.21.15.105:0/3693870526 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fca98038330 msgr2=0x7fca9803a7f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T02:42:03.778 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:02.156+0000 7fca9f7fe700 1 --2- 172.21.15.105:0/3693870526 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fca98038330 0x7fca9803a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:03.778 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:03.157+0000 7fca9f7fe700 1 -- 172.21.15.105:0/3693870526 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 7) v1 ==== 45177+0+0 (secure 0 0 0) 0x7fcaa404e100 con 0x7fcab0107040 2024-04-06T02:42:03.778 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:03.157+0000 7fca9f7fe700 1 --2- 172.21.15.105:0/3693870526 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fca9803bab0 0x7fca9803dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:03.778 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:03.157+0000 7fca9f7fe700 1 -- 172.21.15.105:0/3693870526 --> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcab010a690 con 0x7fca9803bab0 2024-04-06T02:42:03.778 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:03.158+0000 7fcaadd9b700 1 --2- 172.21.15.105:0/3693870526 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fca9803bab0 0x7fca9803dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:03.778 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:03.158+0000 7fcaadd9b700 1 --2- 172.21.15.105:0/3693870526 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fca9803bab0 0x7fca9803dea0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fcaa0003a10 tx=0x7fcaa00092b0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:03.778 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:03.159+0000 7fca9f7fe700 1 -- 172.21.15.105:0/3693870526 <== mgr.14118 v2:172.21.15.105:6800/1394104199 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fcab010a690 con 0x7fca9803bab0 2024-04-06T02:42:03.778 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:03.165+0000 7fcab4ca4700 1 -- 172.21.15.105:0/3693870526 --> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fcab019b4a0 con 0x7fca9803bab0 2024-04-06T02:42:03.778 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:03.165+0000 7fca9f7fe700 1 -- 172.21.15.105:0/3693870526 <== mgr.14118 v2:172.21.15.105:6800/1394104199 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fcab019b4a0 con 0x7fca9803bab0 2024-04-06T02:42:03.778 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:03.165+0000 7fcab4ca4700 1 -- 172.21.15.105:0/3693870526 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fca9803bab0 msgr2=0x7fca9803dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:03.779 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:03.165+0000 7fcab4ca4700 1 --2- 172.21.15.105:0/3693870526 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fca9803bab0 0x7fca9803dea0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fcaa0003a10 tx=0x7fcaa00092b0 comp rx=0 tx=0).stop 2024-04-06T02:42:03.779 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:03.165+0000 7fcab4ca4700 1 -- 172.21.15.105:0/3693870526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcab0107040 msgr2=0x7fcab019a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:03.779 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:03.165+0000 7fcab4ca4700 1 --2- 172.21.15.105:0/3693870526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcab0107040 0x7fcab019a160 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcaa40119e0 tx=0x7fcaa4011ac0 comp rx=0 tx=0).stop 2024-04-06T02:42:03.779 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:03.165+0000 7fcab4ca4700 1 -- 172.21.15.105:0/3693870526 shutdown_connections 2024-04-06T02:42:03.779 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:03.165+0000 7fcab4ca4700 1 --2- 172.21.15.105:0/3693870526 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fca9803bab0 0x7fca9803dea0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:03.779 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:03.165+0000 7fcab4ca4700 1 --2- 172.21.15.105:0/3693870526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcab0107040 0x7fcab019a160 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:03.779 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:03.165+0000 7fcab4ca4700 1 --2- 172.21.15.105:0/3693870526 >> [v2:172.21.15.105:6800/3245193613,v1:172.21.15.105:6801/3245193613] conn(0x7fca98038330 0x7fca9803a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:03.779 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:03.165+0000 7fcab4ca4700 1 -- 172.21.15.105:0/3693870526 >> 172.21.15.105:0/3693870526 conn(0x7fcab01009f0 msgr2=0x7fcab0101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:03.779 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:03.165+0000 7fcab4ca4700 1 -- 172.21.15.105:0/3693870526 shutdown_connections 2024-04-06T02:42:03.779 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:03.166+0000 7fcab4ca4700 1 -- 172.21.15.105:0/3693870526 wait complete. 2024-04-06T02:42:03.779 INFO:teuthology.orchestra.run.smithi105.stdout:mgr epoch 5 is available 2024-04-06T02:42:03.779 INFO:teuthology.orchestra.run.smithi105.stdout:Setting orchestrator backend to cephadm... 2024-04-06T02:42:04.392 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:04 smithi105 ceph-mon[29228]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-06T02:42:04.392 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:04 smithi105 ceph-mon[29228]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-06T02:42:04.392 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:04 smithi105 ceph-mon[29228]: [06/Apr/2024:02:42:03] ENGINE Bus STARTING 2024-04-06T02:42:04.392 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:04 smithi105 ceph-mon[29228]: [06/Apr/2024:02:42:03] ENGINE Serving on http://172.21.15.105:8765 2024-04-06T02:42:04.392 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:04 smithi105 ceph-mon[29228]: [06/Apr/2024:02:42:03] ENGINE Serving on https://172.21.15.105:7150 2024-04-06T02:42:04.392 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:04 smithi105 ceph-mon[29228]: [06/Apr/2024:02:42:03] ENGINE Bus STARTED 2024-04-06T02:42:04.392 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:04 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:42:04.927 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.198+0000 7ff07e78e700 1 Processor -- start 2024-04-06T02:42:04.928 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.198+0000 7ff07e78e700 1 -- start start 2024-04-06T02:42:04.928 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.198+0000 7ff07e78e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff078106580 0x7ff0781069a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:04.928 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.198+0000 7ff07e78e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff078106f70 con 0x7ff078106580 2024-04-06T02:42:04.928 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.198+0000 7ff077fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff078106580 0x7ff0781069a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:04.928 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.198+0000 7ff077fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff078106580 0x7ff0781069a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43850/0 (socket says 172.21.15.105:43850) 2024-04-06T02:42:04.928 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.198+0000 7ff077fff700 1 -- 172.21.15.105:0/2626250952 learned_addr learned my addr 172.21.15.105:0/2626250952 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:04.928 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.198+0000 7ff077fff700 1 -- 172.21.15.105:0/2626250952 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff078107780 con 0x7ff078106580 2024-04-06T02:42:04.928 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.198+0000 7ff077fff700 1 --2- 172.21.15.105:0/2626250952 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff078106580 0x7ff0781069a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff0680098a0 tx=0x7ff068009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97788bfff87edeea server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:04.928 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.199+0000 7ff076ffd700 1 -- 172.21.15.105:0/2626250952 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff068004180 con 0x7ff078106580 2024-04-06T02:42:04.928 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.199+0000 7ff076ffd700 1 -- 172.21.15.105:0/2626250952 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff0680042e0 con 0x7ff078106580 2024-04-06T02:42:04.928 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.199+0000 7ff076ffd700 1 -- 172.21.15.105:0/2626250952 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff068011a50 con 0x7ff078106580 2024-04-06T02:42:04.928 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.199+0000 7ff07e78e700 1 -- 172.21.15.105:0/2626250952 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff078106580 msgr2=0x7ff0781069a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:04.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.199+0000 7ff07e78e700 1 --2- 172.21.15.105:0/2626250952 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff078106580 0x7ff0781069a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff0680098a0 tx=0x7ff068009bb0 comp rx=0 tx=0).stop 2024-04-06T02:42:04.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.199+0000 7ff07e78e700 1 -- 172.21.15.105:0/2626250952 shutdown_connections 2024-04-06T02:42:04.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.199+0000 7ff07e78e700 1 --2- 172.21.15.105:0/2626250952 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff078106580 0x7ff0781069a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:04.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.199+0000 7ff07e78e700 1 -- 172.21.15.105:0/2626250952 >> 172.21.15.105:0/2626250952 conn(0x7ff078101b00 msgr2=0x7ff078103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:04.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.199+0000 7ff07e78e700 1 -- 172.21.15.105:0/2626250952 shutdown_connections 2024-04-06T02:42:04.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.199+0000 7ff07e78e700 1 -- 172.21.15.105:0/2626250952 wait complete. 2024-04-06T02:42:04.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.200+0000 7ff07e78e700 1 Processor -- start 2024-04-06T02:42:04.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.200+0000 7ff07e78e700 1 -- start start 2024-04-06T02:42:04.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.200+0000 7ff07e78e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff078106580 0x7ff07819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:04.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.200+0000 7ff07e78e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07819a780 con 0x7ff078106580 2024-04-06T02:42:04.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.200+0000 7ff077fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff078106580 0x7ff07819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:04.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.200+0000 7ff077fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff078106580 0x7ff07819a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43858/0 (socket says 172.21.15.105:43858) 2024-04-06T02:42:04.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.200+0000 7ff077fff700 1 -- 172.21.15.105:0/418586496 learned_addr learned my addr 172.21.15.105:0/418586496 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:04.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.201+0000 7ff077fff700 1 -- 172.21.15.105:0/418586496 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff068009550 con 0x7ff078106580 2024-04-06T02:42:04.930 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.201+0000 7ff077fff700 1 --2- 172.21.15.105:0/418586496 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff078106580 0x7ff07819a240 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff068009520 tx=0x7ff068011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:04.930 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.201+0000 7ff0757fa700 1 -- 172.21.15.105:0/418586496 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff068003a80 con 0x7ff078106580 2024-04-06T02:42:04.930 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.201+0000 7ff0757fa700 1 -- 172.21.15.105:0/418586496 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff068003be0 con 0x7ff078106580 2024-04-06T02:42:04.930 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.201+0000 7ff07e78e700 1 -- 172.21.15.105:0/418586496 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff07819a980 con 0x7ff078106580 2024-04-06T02:42:04.930 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.201+0000 7ff0757fa700 1 -- 172.21.15.105:0/418586496 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0680197b0 con 0x7ff078106580 2024-04-06T02:42:04.930 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.201+0000 7ff07e78e700 1 -- 172.21.15.105:0/418586496 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff07819ae20 con 0x7ff078106580 2024-04-06T02:42:04.930 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.202+0000 7ff0757fa700 1 -- 172.21.15.105:0/418586496 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 7) v1 ==== 45177+0+0 (secure 0 0 0) 0x7ff068019ab0 con 0x7ff078106580 2024-04-06T02:42:04.930 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.202+0000 7ff07e78e700 1 -- 172.21.15.105:0/418586496 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0780623c0 con 0x7ff078106580 2024-04-06T02:42:04.930 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.202+0000 7ff0757fa700 1 --2- 172.21.15.105:0/418586496 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7ff0600381a0 0x7ff06003a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:04.930 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.202+0000 7ff0757fa700 1 -- 172.21.15.105:0/418586496 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff06804c3a0 con 0x7ff078106580 2024-04-06T02:42:04.930 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.202+0000 7ff0777fe700 1 --2- 172.21.15.105:0/418586496 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7ff0600381a0 0x7ff06003a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:04.930 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.203+0000 7ff0777fe700 1 --2- 172.21.15.105:0/418586496 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7ff0600381a0 0x7ff06003a660 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff06c006fd0 tx=0x7ff06c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:04.931 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.207+0000 7ff0757fa700 1 -- 172.21.15.105:0/418586496 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff06802b750 con 0x7ff078106580 2024-04-06T02:42:04.931 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.209+0000 7ff0757fa700 1 -- 172.21.15.105:0/418586496 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7ff06802dce0 con 0x7ff078106580 2024-04-06T02:42:04.931 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.407+0000 7ff07e78e700 1 -- 172.21.15.105:0/418586496 --> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7ff07810a980 con 0x7ff0600381a0 2024-04-06T02:42:04.931 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.415+0000 7ff0757fa700 1 -- 172.21.15.105:0/418586496 <== mgr.14118 v2:172.21.15.105:6800/1394104199 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff07810a980 con 0x7ff0600381a0 2024-04-06T02:42:04.931 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.417+0000 7ff07e78e700 1 -- 172.21.15.105:0/418586496 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7ff0600381a0 msgr2=0x7ff06003a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:04.931 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.417+0000 7ff07e78e700 1 --2- 172.21.15.105:0/418586496 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7ff0600381a0 0x7ff06003a660 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff06c006fd0 tx=0x7ff06c006e40 comp rx=0 tx=0).stop 2024-04-06T02:42:04.931 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.417+0000 7ff07e78e700 1 -- 172.21.15.105:0/418586496 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff078106580 msgr2=0x7ff07819a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:04.931 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.417+0000 7ff07e78e700 1 --2- 172.21.15.105:0/418586496 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff078106580 0x7ff07819a240 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff068009520 tx=0x7ff068011ab0 comp rx=0 tx=0).stop 2024-04-06T02:42:04.931 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.418+0000 7ff07e78e700 1 -- 172.21.15.105:0/418586496 shutdown_connections 2024-04-06T02:42:04.931 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.418+0000 7ff07e78e700 1 --2- 172.21.15.105:0/418586496 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7ff0600381a0 0x7ff06003a660 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:04.931 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.418+0000 7ff07e78e700 1 --2- 172.21.15.105:0/418586496 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff078106580 0x7ff07819a240 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:04.931 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.418+0000 7ff07e78e700 1 -- 172.21.15.105:0/418586496 >> 172.21.15.105:0/418586496 conn(0x7ff078101b00 msgr2=0x7ff078102930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:04.931 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.418+0000 7ff07e78e700 1 -- 172.21.15.105:0/418586496 shutdown_connections 2024-04-06T02:42:04.931 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:04.418+0000 7ff07e78e700 1 -- 172.21.15.105:0/418586496 wait complete. 2024-04-06T02:42:05.304 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:05 smithi105 ceph-mon[29228]: mgrmap e8: smithi105.vqqcpx(active, since 2s) 2024-04-06T02:42:05.304 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:05 smithi105 ceph-mon[29228]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:42:05.305 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:05 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:05.305 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:05 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:42:05.983 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-06T02:42:05.984 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.320+0000 7f9702015700 1 Processor -- start 2024-04-06T02:42:05.984 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.320+0000 7f9702015700 1 -- start start 2024-04-06T02:42:05.984 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.320+0000 7f9702015700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f96fc106560 0x7f96fc106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:05.984 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.320+0000 7f9702015700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc106f50 con 0x7f96fc106560 2024-04-06T02:42:05.984 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.320+0000 7f96fb7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f96fc106560 0x7f96fc106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:05.984 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.320+0000 7f96fb7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f96fc106560 0x7f96fc106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43866/0 (socket says 172.21.15.105:43866) 2024-04-06T02:42:05.984 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.320+0000 7f96fb7fe700 1 -- 172.21.15.105:0/3356412154 learned_addr learned my addr 172.21.15.105:0/3356412154 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:05.984 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.321+0000 7f96fb7fe700 1 -- 172.21.15.105:0/3356412154 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96fc107760 con 0x7f96fc106560 2024-04-06T02:42:05.984 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.321+0000 7f96fb7fe700 1 --2- 172.21.15.105:0/3356412154 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f96fc106560 0x7f96fc106980 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f96ec0098a0 tx=0x7f96ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cad1f45290ba37e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:05.984 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.321+0000 7f96fa7fc700 1 -- 172.21.15.105:0/3356412154 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96ec004180 con 0x7f96fc106560 2024-04-06T02:42:05.984 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.321+0000 7f96fa7fc700 1 -- 172.21.15.105:0/3356412154 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f96ec0042e0 con 0x7f96fc106560 2024-04-06T02:42:05.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.321+0000 7f96fa7fc700 1 -- 172.21.15.105:0/3356412154 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96ec011a50 con 0x7f96fc106560 2024-04-06T02:42:05.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.322+0000 7f9702015700 1 -- 172.21.15.105:0/3356412154 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f96fc106560 msgr2=0x7f96fc106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:05.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.322+0000 7f9702015700 1 --2- 172.21.15.105:0/3356412154 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f96fc106560 0x7f96fc106980 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f96ec0098a0 tx=0x7f96ec009bb0 comp rx=0 tx=0).stop 2024-04-06T02:42:05.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.322+0000 7f9702015700 1 -- 172.21.15.105:0/3356412154 shutdown_connections 2024-04-06T02:42:05.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.322+0000 7f9702015700 1 --2- 172.21.15.105:0/3356412154 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f96fc106560 0x7f96fc106980 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:05.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.322+0000 7f9702015700 1 -- 172.21.15.105:0/3356412154 >> 172.21.15.105:0/3356412154 conn(0x7f96fc101b40 msgr2=0x7f96fc103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:05.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.322+0000 7f9702015700 1 -- 172.21.15.105:0/3356412154 shutdown_connections 2024-04-06T02:42:05.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.322+0000 7f9702015700 1 -- 172.21.15.105:0/3356412154 wait complete. 2024-04-06T02:42:05.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.322+0000 7f9702015700 1 Processor -- start 2024-04-06T02:42:05.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.322+0000 7f9702015700 1 -- start start 2024-04-06T02:42:05.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.323+0000 7f9702015700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f96fc106560 0x7f96fc19a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:05.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.323+0000 7f9702015700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc19a730 con 0x7f96fc106560 2024-04-06T02:42:05.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.323+0000 7f96fb7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f96fc106560 0x7f96fc19a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:05.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.323+0000 7f96fb7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f96fc106560 0x7f96fc19a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43874/0 (socket says 172.21.15.105:43874) 2024-04-06T02:42:05.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.323+0000 7f96fb7fe700 1 -- 172.21.15.105:0/3765093374 learned_addr learned my addr 172.21.15.105:0/3765093374 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:05.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.323+0000 7f96fb7fe700 1 -- 172.21.15.105:0/3765093374 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ec009550 con 0x7f96fc106560 2024-04-06T02:42:05.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.323+0000 7f96fb7fe700 1 --2- 172.21.15.105:0/3765093374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f96fc106560 0x7f96fc19a1f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f96ec003de0 tx=0x7f96ec011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:05.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.323+0000 7f96f8ff9700 1 -- 172.21.15.105:0/3765093374 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96ec003930 con 0x7f96fc106560 2024-04-06T02:42:05.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.323+0000 7f96f8ff9700 1 -- 172.21.15.105:0/3765093374 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f96ec003a90 con 0x7f96fc106560 2024-04-06T02:42:05.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.323+0000 7f9702015700 1 -- 172.21.15.105:0/3765093374 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96fc19a930 con 0x7f96fc106560 2024-04-06T02:42:05.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.323+0000 7f96f8ff9700 1 -- 172.21.15.105:0/3765093374 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96ec0196c0 con 0x7f96fc106560 2024-04-06T02:42:05.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.323+0000 7f9702015700 1 -- 172.21.15.105:0/3765093374 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96fc19add0 con 0x7f96fc106560 2024-04-06T02:42:05.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.324+0000 7f96f8ff9700 1 -- 172.21.15.105:0/3765093374 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f96ec019930 con 0x7f96fc106560 2024-04-06T02:42:05.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.324+0000 7f96f8ff9700 1 --2- 172.21.15.105:0/3765093374 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f96e4038370 0x7f96e403a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:05.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.324+0000 7f96f8ff9700 1 -- 172.21.15.105:0/3765093374 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f96ec04c3a0 con 0x7f96fc106560 2024-04-06T02:42:05.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.324+0000 7f96faffd700 1 --2- 172.21.15.105:0/3765093374 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f96e4038370 0x7f96e403a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:05.987 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.324+0000 7f9702015700 1 -- 172.21.15.105:0/3765093374 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96fc0623c0 con 0x7f96fc106560 2024-04-06T02:42:05.987 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.325+0000 7f96faffd700 1 --2- 172.21.15.105:0/3765093374 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f96e4038370 0x7f96e403a830 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f96f0006fd0 tx=0x7f96f0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:05.987 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.329+0000 7f96f8ff9700 1 -- 172.21.15.105:0/3765093374 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f96ec019c30 con 0x7f96fc106560 2024-04-06T02:42:05.987 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.518+0000 7f9702015700 1 -- 172.21.15.105:0/3765093374 --> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f96fc103010 con 0x7f96e4038370 2024-04-06T02:42:05.987 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.519+0000 7f96f8ff9700 1 -- 172.21.15.105:0/3765093374 <== mgr.14118 v2:172.21.15.105:6800/1394104199 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f96fc103010 con 0x7f96e4038370 2024-04-06T02:42:05.987 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.521+0000 7f9702015700 1 -- 172.21.15.105:0/3765093374 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f96e4038370 msgr2=0x7f96e403a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:05.987 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.521+0000 7f9702015700 1 --2- 172.21.15.105:0/3765093374 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f96e4038370 0x7f96e403a830 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f96f0006fd0 tx=0x7f96f0006e40 comp rx=0 tx=0).stop 2024-04-06T02:42:05.987 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.521+0000 7f9702015700 1 -- 172.21.15.105:0/3765093374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f96fc106560 msgr2=0x7f96fc19a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:05.987 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.521+0000 7f9702015700 1 --2- 172.21.15.105:0/3765093374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f96fc106560 0x7f96fc19a1f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f96ec003de0 tx=0x7f96ec011ab0 comp rx=0 tx=0).stop 2024-04-06T02:42:05.987 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.521+0000 7f9702015700 1 -- 172.21.15.105:0/3765093374 shutdown_connections 2024-04-06T02:42:05.987 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.521+0000 7f9702015700 1 --2- 172.21.15.105:0/3765093374 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f96e4038370 0x7f96e403a830 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:05.987 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.521+0000 7f9702015700 1 --2- 172.21.15.105:0/3765093374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f96fc106560 0x7f96fc19a1f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:05.987 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.521+0000 7f9702015700 1 -- 172.21.15.105:0/3765093374 >> 172.21.15.105:0/3765093374 conn(0x7f96fc101b40 msgr2=0x7f96fc102900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:05.987 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.521+0000 7f9702015700 1 -- 172.21.15.105:0/3765093374 shutdown_connections 2024-04-06T02:42:05.988 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:05.521+0000 7f9702015700 1 -- 172.21.15.105:0/3765093374 wait complete. 2024-04-06T02:42:05.988 INFO:teuthology.orchestra.run.smithi105.stdout:Generating ssh key... 2024-04-06T02:42:06.379 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:06 smithi105 ceph-mon[29228]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:42:07.715 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.419+0000 7f6479c2e700 1 Processor -- start 2024-04-06T02:42:07.715 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.419+0000 7f6479c2e700 1 -- start start 2024-04-06T02:42:07.715 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.419+0000 7f6479c2e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6474104840 0x7f6474104c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:07.715 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.419+0000 7f6479c2e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6474105230 con 0x7f6474104840 2024-04-06T02:42:07.716 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.420+0000 7f64737fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6474104840 0x7f6474104c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:07.716 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.420+0000 7f64737fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6474104840 0x7f6474104c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43882/0 (socket says 172.21.15.105:43882) 2024-04-06T02:42:07.716 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.420+0000 7f64737fe700 1 -- 172.21.15.105:0/3517483320 learned_addr learned my addr 172.21.15.105:0/3517483320 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:07.716 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.420+0000 7f64737fe700 1 -- 172.21.15.105:0/3517483320 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6474105370 con 0x7f6474104840 2024-04-06T02:42:07.716 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.420+0000 7f64737fe700 1 --2- 172.21.15.105:0/3517483320 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6474104840 0x7f6474104c60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f64640098a0 tx=0x7f6464009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69e24481e478086f server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:07.716 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.420+0000 7f6472ffd700 1 -- 172.21.15.105:0/3517483320 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6464004180 con 0x7f6474104840 2024-04-06T02:42:07.716 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.420+0000 7f6472ffd700 1 -- 172.21.15.105:0/3517483320 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f64640042e0 con 0x7f6474104840 2024-04-06T02:42:07.716 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.420+0000 7f6472ffd700 1 -- 172.21.15.105:0/3517483320 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6464011a50 con 0x7f6474104840 2024-04-06T02:42:07.716 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.421+0000 7f6479c2e700 1 -- 172.21.15.105:0/3517483320 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6474104840 msgr2=0x7f6474104c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:07.716 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.421+0000 7f6479c2e700 1 --2- 172.21.15.105:0/3517483320 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6474104840 0x7f6474104c60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f64640098a0 tx=0x7f6464009bb0 comp rx=0 tx=0).stop 2024-04-06T02:42:07.716 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.421+0000 7f6479c2e700 1 -- 172.21.15.105:0/3517483320 shutdown_connections 2024-04-06T02:42:07.716 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.421+0000 7f6479c2e700 1 --2- 172.21.15.105:0/3517483320 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6474104840 0x7f6474104c60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:07.716 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.421+0000 7f6479c2e700 1 -- 172.21.15.105:0/3517483320 >> 172.21.15.105:0/3517483320 conn(0x7f6474100190 msgr2=0x7f64741025b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:07.717 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.421+0000 7f6479c2e700 1 -- 172.21.15.105:0/3517483320 shutdown_connections 2024-04-06T02:42:07.717 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.421+0000 7f6479c2e700 1 -- 172.21.15.105:0/3517483320 wait complete. 2024-04-06T02:42:07.717 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.422+0000 7f6479c2e700 1 Processor -- start 2024-04-06T02:42:07.717 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.422+0000 7f6479c2e700 1 -- start start 2024-04-06T02:42:07.717 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.422+0000 7f6479c2e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6474104840 0x7f647419a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:07.717 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.422+0000 7f6479c2e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647419a730 con 0x7f6474104840 2024-04-06T02:42:07.717 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.422+0000 7f64737fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6474104840 0x7f647419a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:07.717 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.422+0000 7f64737fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6474104840 0x7f647419a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43890/0 (socket says 172.21.15.105:43890) 2024-04-06T02:42:07.717 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.422+0000 7f64737fe700 1 -- 172.21.15.105:0/2795666960 learned_addr learned my addr 172.21.15.105:0/2795666960 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:07.717 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.422+0000 7f64737fe700 1 -- 172.21.15.105:0/2795666960 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6464009550 con 0x7f6474104840 2024-04-06T02:42:07.717 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.422+0000 7f64737fe700 1 --2- 172.21.15.105:0/2795666960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6474104840 0x7f647419a1f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6464003de0 tx=0x7f6464011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:07.717 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.422+0000 7f64717fa700 1 -- 172.21.15.105:0/2795666960 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6464003930 con 0x7f6474104840 2024-04-06T02:42:07.717 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.422+0000 7f64717fa700 1 -- 172.21.15.105:0/2795666960 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6464003a90 con 0x7f6474104840 2024-04-06T02:42:07.717 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.422+0000 7f6479c2e700 1 -- 172.21.15.105:0/2795666960 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f647419a930 con 0x7f6474104840 2024-04-06T02:42:07.718 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.422+0000 7f64717fa700 1 -- 172.21.15.105:0/2795666960 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64640196c0 con 0x7f6474104840 2024-04-06T02:42:07.718 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.422+0000 7f6479c2e700 1 -- 172.21.15.105:0/2795666960 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f647419add0 con 0x7f6474104840 2024-04-06T02:42:07.718 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.423+0000 7f64717fa700 1 -- 172.21.15.105:0/2795666960 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f64640199a0 con 0x7f6474104840 2024-04-06T02:42:07.718 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.423+0000 7f6479c2e700 1 -- 172.21.15.105:0/2795666960 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64740623c0 con 0x7f6474104840 2024-04-06T02:42:07.718 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.424+0000 7f64717fa700 1 --2- 172.21.15.105:0/2795666960 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f6460038320 0x7f646003a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:07.718 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.424+0000 7f64717fa700 1 -- 172.21.15.105:0/2795666960 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f646404c3e0 con 0x7f6474104840 2024-04-06T02:42:07.718 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.424+0000 7f646bfff700 1 --2- 172.21.15.105:0/2795666960 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f6460038320 0x7f646003a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:07.718 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.424+0000 7f646bfff700 1 --2- 172.21.15.105:0/2795666960 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f6460038320 0x7f646003a7e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f645c006fd0 tx=0x7f645c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:07.718 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.428+0000 7f64717fa700 1 -- 172.21.15.105:0/2795666960 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6464023370 con 0x7f6474104840 2024-04-06T02:42:07.718 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:06.617+0000 7f6479c2e700 1 -- 172.21.15.105:0/2795666960 --> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f6474102480 con 0x7f6460038320 2024-04-06T02:42:07.718 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:07.130+0000 7f64717fa700 1 -- 172.21.15.105:0/2795666960 <== mgr.14118 v2:172.21.15.105:6800/1394104199 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6474102480 con 0x7f6460038320 2024-04-06T02:42:07.718 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:07.132+0000 7f6479c2e700 1 -- 172.21.15.105:0/2795666960 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f6460038320 msgr2=0x7f646003a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:07.718 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:07.132+0000 7f6479c2e700 1 --2- 172.21.15.105:0/2795666960 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f6460038320 0x7f646003a7e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f645c006fd0 tx=0x7f645c006e40 comp rx=0 tx=0).stop 2024-04-06T02:42:07.719 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:07.132+0000 7f6479c2e700 1 -- 172.21.15.105:0/2795666960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6474104840 msgr2=0x7f647419a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:07.719 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:07.132+0000 7f6479c2e700 1 --2- 172.21.15.105:0/2795666960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6474104840 0x7f647419a1f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6464003de0 tx=0x7f6464011ab0 comp rx=0 tx=0).stop 2024-04-06T02:42:07.719 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:07.133+0000 7f6479c2e700 1 -- 172.21.15.105:0/2795666960 shutdown_connections 2024-04-06T02:42:07.719 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:07.133+0000 7f6479c2e700 1 --2- 172.21.15.105:0/2795666960 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f6460038320 0x7f646003a7e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:07.719 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:07.133+0000 7f6479c2e700 1 --2- 172.21.15.105:0/2795666960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6474104840 0x7f647419a1f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:07.719 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:07.133+0000 7f6479c2e700 1 -- 172.21.15.105:0/2795666960 >> 172.21.15.105:0/2795666960 conn(0x7f6474100190 msgr2=0x7f6474101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:07.719 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:07.133+0000 7f6479c2e700 1 -- 172.21.15.105:0/2795666960 shutdown_connections 2024-04-06T02:42:07.719 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:07.133+0000 7f6479c2e700 1 -- 172.21.15.105:0/2795666960 wait complete. 2024-04-06T02:42:08.317 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:08 smithi105 ceph-mon[29228]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:42:08.317 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:08 smithi105 ceph-mon[29228]: Generating ssh key... 2024-04-06T02:42:08.317 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:08 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:08.317 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:08 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:08.897 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMS5W7QPe7kQnHN+Ai3JPQqf38tfmBhbG/l2unsLPQjKI593vabhr3YyySPU36wF7C3aq0KFas0BCZvPXHpZ41WO0SEjRPhLLNqQbopbFXZr4n94AMChMlg1ovkVKQdIV6fjSMr0foGoS1rRzFoYWD3wJGRDuw5WrQQTanjE6Lbz9LWeniCFbT8TUXIUpDGURxSDLCmr0rYgwTOC+pQizeaEe4sVrhARzlMI9J7Nu5oQ22iFz0yDhkPLhDR70kr2I7lwYZ7FmdHhUynF/XsttuagDwL1bmdHAIw5Y7jljBJvUyZzgXPkt1lMkLaEZuL1qfgn/1fKHawaMuCeY8K2GCyzW7mv3wtJhUrw/jMRuQR5x8mRmXQmzT1R06faa/bwCsQH7PDXak6yMRrGXzom0Ja80miJI9WD+Cr/3ne9cmypy80shjIX2KgvTbRuDfwUjqs1Q0D2+ebArUebGWacnCACg27WXnDbB7ONwaRn8l+W6Omdhti9ZAOSfOly2oV2c= ceph-130df578-f3bf-11ee-b649-cb9ed24678a4 2024-04-06T02:42:08.897 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.135+0000 7f32e083a700 1 Processor -- start 2024-04-06T02:42:08.897 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.135+0000 7f32e083a700 1 -- start start 2024-04-06T02:42:08.897 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.135+0000 7f32e083a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32dc104850 0x7f32dc104c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:08.897 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.135+0000 7f32e083a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32dc105240 con 0x7f32dc104850 2024-04-06T02:42:08.897 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.135+0000 7f32da59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32dc104850 0x7f32dc104c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:08.898 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.135+0000 7f32da59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32dc104850 0x7f32dc104c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43902/0 (socket says 172.21.15.105:43902) 2024-04-06T02:42:08.898 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.135+0000 7f32da59c700 1 -- 172.21.15.105:0/545732059 learned_addr learned my addr 172.21.15.105:0/545732059 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:08.898 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.135+0000 7f32da59c700 1 -- 172.21.15.105:0/545732059 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32dc105380 con 0x7f32dc104850 2024-04-06T02:42:08.898 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.136+0000 7f32da59c700 1 --2- 172.21.15.105:0/545732059 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32dc104850 0x7f32dc104c70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f32d40098a0 tx=0x7f32d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1ae5223706f60be server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:08.898 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.136+0000 7f32d959a700 1 -- 172.21.15.105:0/545732059 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32d4004180 con 0x7f32dc104850 2024-04-06T02:42:08.898 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.136+0000 7f32d959a700 1 -- 172.21.15.105:0/545732059 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f32d40042e0 con 0x7f32dc104850 2024-04-06T02:42:08.898 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.136+0000 7f32d959a700 1 -- 172.21.15.105:0/545732059 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32d4011a50 con 0x7f32dc104850 2024-04-06T02:42:08.898 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.136+0000 7f32e083a700 1 -- 172.21.15.105:0/545732059 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32dc104850 msgr2=0x7f32dc104c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:08.898 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.136+0000 7f32e083a700 1 --2- 172.21.15.105:0/545732059 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32dc104850 0x7f32dc104c70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f32d40098a0 tx=0x7f32d4009bb0 comp rx=0 tx=0).stop 2024-04-06T02:42:08.898 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.136+0000 7f32e083a700 1 -- 172.21.15.105:0/545732059 shutdown_connections 2024-04-06T02:42:08.898 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.136+0000 7f32e083a700 1 --2- 172.21.15.105:0/545732059 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32dc104850 0x7f32dc104c70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:08.898 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.136+0000 7f32e083a700 1 -- 172.21.15.105:0/545732059 >> 172.21.15.105:0/545732059 conn(0x7f32dc1001a0 msgr2=0x7f32dc1025c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:08.898 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.136+0000 7f32e083a700 1 -- 172.21.15.105:0/545732059 shutdown_connections 2024-04-06T02:42:08.899 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.136+0000 7f32e083a700 1 -- 172.21.15.105:0/545732059 wait complete. 2024-04-06T02:42:08.899 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.137+0000 7f32e083a700 1 Processor -- start 2024-04-06T02:42:08.899 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.137+0000 7f32e083a700 1 -- start start 2024-04-06T02:42:08.899 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.137+0000 7f32e083a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32dc104850 0x7f32dc1071e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:08.899 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.137+0000 7f32e083a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32dc107720 con 0x7f32dc104850 2024-04-06T02:42:08.899 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.137+0000 7f32da59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32dc104850 0x7f32dc1071e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:08.899 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.137+0000 7f32da59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32dc104850 0x7f32dc1071e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43918/0 (socket says 172.21.15.105:43918) 2024-04-06T02:42:08.899 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.137+0000 7f32da59c700 1 -- 172.21.15.105:0/3210117137 learned_addr learned my addr 172.21.15.105:0/3210117137 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:08.899 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.138+0000 7f32da59c700 1 -- 172.21.15.105:0/3210117137 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d4009550 con 0x7f32dc104850 2024-04-06T02:42:08.899 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.138+0000 7f32da59c700 1 --2- 172.21.15.105:0/3210117137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32dc104850 0x7f32dc1071e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f32d4003de0 tx=0x7f32d4011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:08.899 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.138+0000 7f32cb7fe700 1 -- 172.21.15.105:0/3210117137 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32d4003930 con 0x7f32dc104850 2024-04-06T02:42:08.899 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.138+0000 7f32e083a700 1 -- 172.21.15.105:0/3210117137 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32dc105830 con 0x7f32dc104850 2024-04-06T02:42:08.899 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.138+0000 7f32cb7fe700 1 -- 172.21.15.105:0/3210117137 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f32d4003a90 con 0x7f32dc104850 2024-04-06T02:42:08.899 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.138+0000 7f32cb7fe700 1 -- 172.21.15.105:0/3210117137 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32d40196c0 con 0x7f32dc104850 2024-04-06T02:42:08.900 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.138+0000 7f32e083a700 1 -- 172.21.15.105:0/3210117137 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32dc105cd0 con 0x7f32dc104850 2024-04-06T02:42:08.900 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.139+0000 7f32cb7fe700 1 -- 172.21.15.105:0/3210117137 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f32d40199a0 con 0x7f32dc104850 2024-04-06T02:42:08.900 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.139+0000 7f32e083a700 1 -- 172.21.15.105:0/3210117137 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32dc0623c0 con 0x7f32dc104850 2024-04-06T02:42:08.900 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.139+0000 7f32cb7fe700 1 --2- 172.21.15.105:0/3210117137 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f32c4038320 0x7f32c403a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:08.900 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.139+0000 7f32cb7fe700 1 -- 172.21.15.105:0/3210117137 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f32d404c3b0 con 0x7f32dc104850 2024-04-06T02:42:08.900 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.139+0000 7f32d9d9b700 1 --2- 172.21.15.105:0/3210117137 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f32c4038320 0x7f32c403a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:08.900 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.140+0000 7f32d9d9b700 1 --2- 172.21.15.105:0/3210117137 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f32c4038320 0x7f32c403a7e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f32d0006fd0 tx=0x7f32d0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:08.900 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.144+0000 7f32cb7fe700 1 -- 172.21.15.105:0/3210117137 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f32d4023370 con 0x7f32dc104850 2024-04-06T02:42:08.900 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.333+0000 7f32e083a700 1 -- 172.21.15.105:0/3210117137 --> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f32dc101690 con 0x7f32c4038320 2024-04-06T02:42:08.900 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.333+0000 7f32cb7fe700 1 -- 172.21.15.105:0/3210117137 <== mgr.14118 v2:172.21.15.105:6800/1394104199 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f32dc101690 con 0x7f32c4038320 2024-04-06T02:42:08.900 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.335+0000 7f32e083a700 1 -- 172.21.15.105:0/3210117137 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f32c4038320 msgr2=0x7f32c403a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:08.900 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.335+0000 7f32e083a700 1 --2- 172.21.15.105:0/3210117137 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f32c4038320 0x7f32c403a7e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f32d0006fd0 tx=0x7f32d0006e40 comp rx=0 tx=0).stop 2024-04-06T02:42:08.900 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.335+0000 7f32e083a700 1 -- 172.21.15.105:0/3210117137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32dc104850 msgr2=0x7f32dc1071e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:08.901 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.335+0000 7f32e083a700 1 --2- 172.21.15.105:0/3210117137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32dc104850 0x7f32dc1071e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f32d4003de0 tx=0x7f32d4011ab0 comp rx=0 tx=0).stop 2024-04-06T02:42:08.901 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.335+0000 7f32e083a700 1 -- 172.21.15.105:0/3210117137 shutdown_connections 2024-04-06T02:42:08.901 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.335+0000 7f32e083a700 1 --2- 172.21.15.105:0/3210117137 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f32c4038320 0x7f32c403a7e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:08.901 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.335+0000 7f32e083a700 1 --2- 172.21.15.105:0/3210117137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32dc104850 0x7f32dc1071e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:08.901 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.336+0000 7f32e083a700 1 -- 172.21.15.105:0/3210117137 >> 172.21.15.105:0/3210117137 conn(0x7f32dc1001a0 msgr2=0x7f32dc100f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:08.901 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.336+0000 7f32e083a700 1 -- 172.21.15.105:0/3210117137 shutdown_connections 2024-04-06T02:42:08.901 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:08.336+0000 7f32e083a700 1 -- 172.21.15.105:0/3210117137 wait complete. 2024-04-06T02:42:08.901 INFO:teuthology.orchestra.run.smithi105.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-06T02:42:08.901 INFO:teuthology.orchestra.run.smithi105.stdout:Adding key to root@localhost authorized_keys... 2024-04-06T02:42:08.901 INFO:teuthology.orchestra.run.smithi105.stdout:Adding host smithi105... 2024-04-06T02:42:09.173 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:09 smithi105 ceph-mon[29228]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:42:10.359 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:10 smithi105 ceph-mon[29228]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi105", "addr": "172.21.15.105", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:42:11.423 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:11 smithi105 ceph-mon[29228]: Deploying cephadm binary to smithi105 2024-04-06T02:42:12.589 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout Added host 'smithi105' with addr '172.21.15.105' 2024-04-06T02:42:12.589 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.319+0000 7f5476d07700 1 Processor -- start 2024-04-06T02:42:12.589 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.319+0000 7f5476d07700 1 -- start start 2024-04-06T02:42:12.589 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.319+0000 7f5476d07700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5470106580 0x7f54701069a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:12.589 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.319+0000 7f5476d07700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5470106f70 con 0x7f5470106580 2024-04-06T02:42:12.589 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.319+0000 7f5474aa3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5470106580 0x7f54701069a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:12.589 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.319+0000 7f5474aa3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5470106580 0x7f54701069a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43934/0 (socket says 172.21.15.105:43934) 2024-04-06T02:42:12.589 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.319+0000 7f5474aa3700 1 -- 172.21.15.105:0/39923738 learned_addr learned my addr 172.21.15.105:0/39923738 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:12.590 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.319+0000 7f5474aa3700 1 -- 172.21.15.105:0/39923738 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5470107780 con 0x7f5470106580 2024-04-06T02:42:12.590 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.319+0000 7f5474aa3700 1 --2- 172.21.15.105:0/39923738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5470106580 0x7f54701069a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f54600098a0 tx=0x7f5460009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8c9d781b3a3a2e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:12.590 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.319+0000 7f546f7fe700 1 -- 172.21.15.105:0/39923738 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5460004180 con 0x7f5470106580 2024-04-06T02:42:12.590 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.320+0000 7f546f7fe700 1 -- 172.21.15.105:0/39923738 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f54600042e0 con 0x7f5470106580 2024-04-06T02:42:12.590 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.320+0000 7f546f7fe700 1 -- 172.21.15.105:0/39923738 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5460011a50 con 0x7f5470106580 2024-04-06T02:42:12.590 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.320+0000 7f5476d07700 1 -- 172.21.15.105:0/39923738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5470106580 msgr2=0x7f54701069a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:12.590 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.320+0000 7f5476d07700 1 --2- 172.21.15.105:0/39923738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5470106580 0x7f54701069a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f54600098a0 tx=0x7f5460009bb0 comp rx=0 tx=0).stop 2024-04-06T02:42:12.590 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.320+0000 7f5476d07700 1 -- 172.21.15.105:0/39923738 shutdown_connections 2024-04-06T02:42:12.590 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.320+0000 7f5476d07700 1 --2- 172.21.15.105:0/39923738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5470106580 0x7f54701069a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:12.590 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.320+0000 7f5476d07700 1 -- 172.21.15.105:0/39923738 >> 172.21.15.105:0/39923738 conn(0x7f5470101b00 msgr2=0x7f5470103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:12.590 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.320+0000 7f5476d07700 1 -- 172.21.15.105:0/39923738 shutdown_connections 2024-04-06T02:42:12.590 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.320+0000 7f5476d07700 1 -- 172.21.15.105:0/39923738 wait complete. 2024-04-06T02:42:12.590 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.321+0000 7f5476d07700 1 Processor -- start 2024-04-06T02:42:12.591 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.321+0000 7f5476d07700 1 -- start start 2024-04-06T02:42:12.591 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.321+0000 7f5476d07700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5470106580 0x7f547019a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:12.591 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.321+0000 7f5476d07700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547019a780 con 0x7f5470106580 2024-04-06T02:42:12.591 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.321+0000 7f5474aa3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5470106580 0x7f547019a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:12.591 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.321+0000 7f5474aa3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5470106580 0x7f547019a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43944/0 (socket says 172.21.15.105:43944) 2024-04-06T02:42:12.591 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.321+0000 7f5474aa3700 1 -- 172.21.15.105:0/1150820968 learned_addr learned my addr 172.21.15.105:0/1150820968 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:12.591 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.321+0000 7f5474aa3700 1 -- 172.21.15.105:0/1150820968 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5460009550 con 0x7f5470106580 2024-04-06T02:42:12.591 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.321+0000 7f5474aa3700 1 --2- 172.21.15.105:0/1150820968 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5470106580 0x7f547019a240 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5460009520 tx=0x7f5460011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:12.591 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.322+0000 7f546dffb700 1 -- 172.21.15.105:0/1150820968 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5460003a80 con 0x7f5470106580 2024-04-06T02:42:12.591 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.322+0000 7f546dffb700 1 -- 172.21.15.105:0/1150820968 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5460003be0 con 0x7f5470106580 2024-04-06T02:42:12.591 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.322+0000 7f546dffb700 1 -- 172.21.15.105:0/1150820968 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54600197b0 con 0x7f5470106580 2024-04-06T02:42:12.591 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.322+0000 7f5476d07700 1 -- 172.21.15.105:0/1150820968 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f547019a980 con 0x7f5470106580 2024-04-06T02:42:12.592 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.322+0000 7f5476d07700 1 -- 172.21.15.105:0/1150820968 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f547019ae20 con 0x7f5470106580 2024-04-06T02:42:12.592 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.323+0000 7f546dffb700 1 -- 172.21.15.105:0/1150820968 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f5460019a20 con 0x7f5470106580 2024-04-06T02:42:12.592 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.323+0000 7f546dffb700 1 --2- 172.21.15.105:0/1150820968 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f5458038320 0x7f545803a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:12.592 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.323+0000 7f546dffb700 1 -- 172.21.15.105:0/1150820968 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f546004c510 con 0x7f5470106580 2024-04-06T02:42:12.592 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.323+0000 7f5476d07700 1 -- 172.21.15.105:0/1150820968 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54700623c0 con 0x7f5470106580 2024-04-06T02:42:12.592 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.323+0000 7f546ffff700 1 --2- 172.21.15.105:0/1150820968 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f5458038320 0x7f545803a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:12.592 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.323+0000 7f546ffff700 1 --2- 172.21.15.105:0/1150820968 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f5458038320 0x7f545803a7e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5464006fd0 tx=0x7f5464006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:12.592 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.328+0000 7f546dffb700 1 -- 172.21.15.105:0/1150820968 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f546002d440 con 0x7f5470106580 2024-04-06T02:42:12.592 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:09.518+0000 7f5476d07700 1 -- 172.21.15.105:0/1150820968 --> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi105", "addr": "172.21.15.105", "target": ["mon-mgr", ""]}) v1 -- 0x7f547010a980 con 0x7f5458038320 2024-04-06T02:42:12.592 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:12.026+0000 7f546dffb700 1 -- 172.21.15.105:0/1150820968 <== mgr.14118 v2:172.21.15.105:6800/1394104199 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f547010a980 con 0x7f5458038320 2024-04-06T02:42:12.592 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:12.029+0000 7f5476d07700 1 -- 172.21.15.105:0/1150820968 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f5458038320 msgr2=0x7f545803a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:12.593 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:12.029+0000 7f5476d07700 1 --2- 172.21.15.105:0/1150820968 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f5458038320 0x7f545803a7e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5464006fd0 tx=0x7f5464006e40 comp rx=0 tx=0).stop 2024-04-06T02:42:12.593 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:12.029+0000 7f5476d07700 1 -- 172.21.15.105:0/1150820968 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5470106580 msgr2=0x7f547019a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:12.593 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:12.029+0000 7f5476d07700 1 --2- 172.21.15.105:0/1150820968 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5470106580 0x7f547019a240 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5460009520 tx=0x7f5460011ab0 comp rx=0 tx=0).stop 2024-04-06T02:42:12.593 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:12.029+0000 7f5476d07700 1 -- 172.21.15.105:0/1150820968 shutdown_connections 2024-04-06T02:42:12.593 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:12.029+0000 7f5476d07700 1 --2- 172.21.15.105:0/1150820968 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f5458038320 0x7f545803a7e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:12.593 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:12.029+0000 7f5476d07700 1 --2- 172.21.15.105:0/1150820968 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5470106580 0x7f547019a240 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:12.593 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:12.029+0000 7f5476d07700 1 -- 172.21.15.105:0/1150820968 >> 172.21.15.105:0/1150820968 conn(0x7f5470101b00 msgr2=0x7f5470102930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:12.593 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:12.029+0000 7f5476d07700 1 -- 172.21.15.105:0/1150820968 shutdown_connections 2024-04-06T02:42:12.593 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:12.029+0000 7f5476d07700 1 -- 172.21.15.105:0/1150820968 wait complete. 2024-04-06T02:42:12.593 INFO:teuthology.orchestra.run.smithi105.stdout:Deploying mon service with default placement... 2024-04-06T02:42:13.267 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:13 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:13.268 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:13 smithi105 ceph-mon[29228]: Added host smithi105 2024-04-06T02:42:13.268 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:13 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:42:14.602 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:14 smithi105 ceph-mon[29228]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:42:14.602 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:14 smithi105 ceph-mon[29228]: Saving service mon spec with placement count:5 2024-04-06T02:42:14.603 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:14 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:14.610 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-06T02:42:14.610 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.041+0000 7fb7c2f4a700 1 Processor -- start 2024-04-06T02:42:14.610 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.041+0000 7fb7c2f4a700 1 -- start start 2024-04-06T02:42:14.610 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.041+0000 7fb7c2f4a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7bc106a30 0x7fb7bc106e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:14.610 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.041+0000 7fb7c2f4a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7bc107420 con 0x7fb7bc106a30 2024-04-06T02:42:14.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.041+0000 7fb7c0ce6700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7bc106a30 0x7fb7bc106e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:14.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.041+0000 7fb7c0ce6700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7bc106a30 0x7fb7bc106e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50264/0 (socket says 172.21.15.105:50264) 2024-04-06T02:42:14.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.041+0000 7fb7c0ce6700 1 -- 172.21.15.105:0/81782111 learned_addr learned my addr 172.21.15.105:0/81782111 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:14.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.041+0000 7fb7c0ce6700 1 -- 172.21.15.105:0/81782111 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7bc105640 con 0x7fb7bc106a30 2024-04-06T02:42:14.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.041+0000 7fb7c0ce6700 1 --2- 172.21.15.105:0/81782111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7bc106a30 0x7fb7bc106e50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4009e80 tx=0x7fb7b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1af8a59088992a96 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:14.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.042+0000 7fb7bb7fe700 1 -- 172.21.15.105:0/81782111 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7b400e070 con 0x7fb7bc106a30 2024-04-06T02:42:14.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.042+0000 7fb7bb7fe700 1 -- 172.21.15.105:0/81782111 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb7b4003e90 con 0x7fb7bc106a30 2024-04-06T02:42:14.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.042+0000 7fb7bb7fe700 1 -- 172.21.15.105:0/81782111 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7b4004380 con 0x7fb7bc106a30 2024-04-06T02:42:14.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.042+0000 7fb7c2f4a700 1 -- 172.21.15.105:0/81782111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7bc106a30 msgr2=0x7fb7bc106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:14.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.042+0000 7fb7c2f4a700 1 --2- 172.21.15.105:0/81782111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7bc106a30 0x7fb7bc106e50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4009e80 tx=0x7fb7b400b330 comp rx=0 tx=0).stop 2024-04-06T02:42:14.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.042+0000 7fb7c2f4a700 1 -- 172.21.15.105:0/81782111 shutdown_connections 2024-04-06T02:42:14.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.042+0000 7fb7c2f4a700 1 --2- 172.21.15.105:0/81782111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7bc106a30 0x7fb7bc106e50 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:14.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.042+0000 7fb7c2f4a700 1 -- 172.21.15.105:0/81782111 >> 172.21.15.105:0/81782111 conn(0x7fb7bc100190 msgr2=0x7fb7bc1025b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:14.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.042+0000 7fb7c2f4a700 1 -- 172.21.15.105:0/81782111 shutdown_connections 2024-04-06T02:42:14.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.042+0000 7fb7c2f4a700 1 -- 172.21.15.105:0/81782111 wait complete. 2024-04-06T02:42:14.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.043+0000 7fb7c2f4a700 1 Processor -- start 2024-04-06T02:42:14.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.043+0000 7fb7c2f4a700 1 -- start start 2024-04-06T02:42:14.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.043+0000 7fb7c2f4a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7bc106a30 0x7fb7bc105090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:14.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.043+0000 7fb7c2f4a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7bc19cf20 con 0x7fb7bc106a30 2024-04-06T02:42:14.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.043+0000 7fb7c0ce6700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7bc106a30 0x7fb7bc105090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:14.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.043+0000 7fb7c0ce6700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7bc106a30 0x7fb7bc105090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50274/0 (socket says 172.21.15.105:50274) 2024-04-06T02:42:14.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.043+0000 7fb7c0ce6700 1 -- 172.21.15.105:0/1748992587 learned_addr learned my addr 172.21.15.105:0/1748992587 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:14.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.043+0000 7fb7c0ce6700 1 -- 172.21.15.105:0/1748992587 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b4009b30 con 0x7fb7bc106a30 2024-04-06T02:42:14.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.043+0000 7fb7c0ce6700 1 --2- 172.21.15.105:0/1748992587 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7bc106a30 0x7fb7bc105090 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4009e50 tx=0x7fb7b4004920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:14.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.043+0000 7fb7b9ffb700 1 -- 172.21.15.105:0/1748992587 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7b400e040 con 0x7fb7bc106a30 2024-04-06T02:42:14.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.044+0000 7fb7b9ffb700 1 -- 172.21.15.105:0/1748992587 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb7b4003cf0 con 0x7fb7bc106a30 2024-04-06T02:42:14.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.044+0000 7fb7c2f4a700 1 -- 172.21.15.105:0/1748992587 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7bc19d060 con 0x7fb7bc106a30 2024-04-06T02:42:14.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.044+0000 7fb7b9ffb700 1 -- 172.21.15.105:0/1748992587 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7b401b770 con 0x7fb7bc106a30 2024-04-06T02:42:14.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.044+0000 7fb7c2f4a700 1 -- 172.21.15.105:0/1748992587 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7bc19d560 con 0x7fb7bc106a30 2024-04-06T02:42:14.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.044+0000 7fb7b9ffb700 1 -- 172.21.15.105:0/1748992587 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fb7b4019040 con 0x7fb7bc106a30 2024-04-06T02:42:14.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.045+0000 7fb7c2f4a700 1 -- 172.21.15.105:0/1748992587 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7bc0623c0 con 0x7fb7bc106a30 2024-04-06T02:42:14.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.045+0000 7fb7b9ffb700 1 --2- 172.21.15.105:0/1748992587 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fb7a8038370 0x7fb7a803a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:14.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.045+0000 7fb7b9ffb700 1 -- 172.21.15.105:0/1748992587 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb7b404bd20 con 0x7fb7bc106a30 2024-04-06T02:42:14.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.045+0000 7fb7bbfff700 1 --2- 172.21.15.105:0/1748992587 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fb7a8038370 0x7fb7a803a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:14.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.045+0000 7fb7bbfff700 1 --2- 172.21.15.105:0/1748992587 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fb7a8038370 0x7fb7a803a830 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac006fd0 tx=0x7fb7ac006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:14.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.049+0000 7fb7b9ffb700 1 -- 172.21.15.105:0/1748992587 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb7b4023820 con 0x7fb7bc106a30 2024-04-06T02:42:14.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.263+0000 7fb7c2f4a700 1 -- 172.21.15.105:0/1748992587 --> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7bc1015f0 con 0x7fb7a8038370 2024-04-06T02:42:14.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.270+0000 7fb7b9ffb700 1 -- 172.21.15.105:0/1748992587 <== mgr.14118 v2:172.21.15.105:6800/1394104199 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fb7bc1015f0 con 0x7fb7a8038370 2024-04-06T02:42:14.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.272+0000 7fb7c2f4a700 1 -- 172.21.15.105:0/1748992587 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fb7a8038370 msgr2=0x7fb7a803a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:14.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.272+0000 7fb7c2f4a700 1 --2- 172.21.15.105:0/1748992587 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fb7a8038370 0x7fb7a803a830 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac006fd0 tx=0x7fb7ac006e40 comp rx=0 tx=0).stop 2024-04-06T02:42:14.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.272+0000 7fb7c2f4a700 1 -- 172.21.15.105:0/1748992587 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7bc106a30 msgr2=0x7fb7bc105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:14.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.272+0000 7fb7c2f4a700 1 --2- 172.21.15.105:0/1748992587 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7bc106a30 0x7fb7bc105090 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4009e50 tx=0x7fb7b4004920 comp rx=0 tx=0).stop 2024-04-06T02:42:14.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.272+0000 7fb7c2f4a700 1 -- 172.21.15.105:0/1748992587 shutdown_connections 2024-04-06T02:42:14.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.272+0000 7fb7c2f4a700 1 --2- 172.21.15.105:0/1748992587 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fb7a8038370 0x7fb7a803a830 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:14.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.272+0000 7fb7c2f4a700 1 --2- 172.21.15.105:0/1748992587 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7bc106a30 0x7fb7bc105090 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:14.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.272+0000 7fb7c2f4a700 1 -- 172.21.15.105:0/1748992587 >> 172.21.15.105:0/1748992587 conn(0x7fb7bc100190 msgr2=0x7fb7bc100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:14.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.273+0000 7fb7c2f4a700 1 -- 172.21.15.105:0/1748992587 shutdown_connections 2024-04-06T02:42:14.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:13.273+0000 7fb7c2f4a700 1 -- 172.21.15.105:0/1748992587 wait complete. 2024-04-06T02:42:14.614 INFO:teuthology.orchestra.run.smithi105.stdout:Deploying mgr service with default placement... 2024-04-06T02:42:15.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:15 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:15.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:15 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:15.951 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-06T02:42:15.951 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.094+0000 7f84339d2700 1 Processor -- start 2024-04-06T02:42:15.951 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.094+0000 7f84339d2700 1 -- start start 2024-04-06T02:42:15.951 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.094+0000 7f84339d2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f842c106570 0x7f842c106990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:15.951 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.094+0000 7f84339d2700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842c106f60 con 0x7f842c106570 2024-04-06T02:42:15.951 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.094+0000 7f843176e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f842c106570 0x7f842c106990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:15.952 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.094+0000 7f843176e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f842c106570 0x7f842c106990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50276/0 (socket says 172.21.15.105:50276) 2024-04-06T02:42:15.952 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.094+0000 7f843176e700 1 -- 172.21.15.105:0/748066673 learned_addr learned my addr 172.21.15.105:0/748066673 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:15.952 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.094+0000 7f843176e700 1 -- 172.21.15.105:0/748066673 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f842c107770 con 0x7f842c106570 2024-04-06T02:42:15.952 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.095+0000 7f843176e700 1 --2- 172.21.15.105:0/748066673 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f842c106570 0x7f842c106990 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f84280098a0 tx=0x7f8428009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29f6fedc05d0edf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:15.952 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.095+0000 7f8423fff700 1 -- 172.21.15.105:0/748066673 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8428004180 con 0x7f842c106570 2024-04-06T02:42:15.952 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.095+0000 7f8423fff700 1 -- 172.21.15.105:0/748066673 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f84280042e0 con 0x7f842c106570 2024-04-06T02:42:15.952 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.095+0000 7f8423fff700 1 -- 172.21.15.105:0/748066673 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8428011af0 con 0x7f842c106570 2024-04-06T02:42:15.952 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.095+0000 7f84339d2700 1 -- 172.21.15.105:0/748066673 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f842c106570 msgr2=0x7f842c106990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:15.952 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.095+0000 7f84339d2700 1 --2- 172.21.15.105:0/748066673 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f842c106570 0x7f842c106990 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f84280098a0 tx=0x7f8428009bb0 comp rx=0 tx=0).stop 2024-04-06T02:42:15.952 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.095+0000 7f84339d2700 1 -- 172.21.15.105:0/748066673 shutdown_connections 2024-04-06T02:42:15.952 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.095+0000 7f84339d2700 1 --2- 172.21.15.105:0/748066673 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f842c106570 0x7f842c106990 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:15.952 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.095+0000 7f84339d2700 1 -- 172.21.15.105:0/748066673 >> 172.21.15.105:0/748066673 conn(0x7f842c101b10 msgr2=0x7f842c103f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:15.952 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.095+0000 7f84339d2700 1 -- 172.21.15.105:0/748066673 shutdown_connections 2024-04-06T02:42:15.952 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.095+0000 7f84339d2700 1 -- 172.21.15.105:0/748066673 wait complete. 2024-04-06T02:42:15.952 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.096+0000 7f84339d2700 1 Processor -- start 2024-04-06T02:42:15.952 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.096+0000 7f84339d2700 1 -- start start 2024-04-06T02:42:15.953 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.096+0000 7f84339d2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f842c106570 0x7f842c19a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:15.953 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.096+0000 7f84339d2700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842c19a7b0 con 0x7f842c106570 2024-04-06T02:42:15.953 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.096+0000 7f843176e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f842c106570 0x7f842c19a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:15.953 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.096+0000 7f843176e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f842c106570 0x7f842c19a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50286/0 (socket says 172.21.15.105:50286) 2024-04-06T02:42:15.953 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.096+0000 7f843176e700 1 -- 172.21.15.105:0/3597214779 learned_addr learned my addr 172.21.15.105:0/3597214779 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:15.953 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.097+0000 7f843176e700 1 -- 172.21.15.105:0/3597214779 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8428009550 con 0x7f842c106570 2024-04-06T02:42:15.953 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.097+0000 7f843176e700 1 --2- 172.21.15.105:0/3597214779 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f842c106570 0x7f842c19a270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8428011670 tx=0x7f8428011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:15.953 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.097+0000 7f84227fc700 1 -- 172.21.15.105:0/3597214779 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84280039c0 con 0x7f842c106570 2024-04-06T02:42:15.953 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.097+0000 7f84227fc700 1 -- 172.21.15.105:0/3597214779 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8428003b20 con 0x7f842c106570 2024-04-06T02:42:15.953 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.097+0000 7f84339d2700 1 -- 172.21.15.105:0/3597214779 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f842c19a9b0 con 0x7f842c106570 2024-04-06T02:42:15.953 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.097+0000 7f84227fc700 1 -- 172.21.15.105:0/3597214779 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84280247f0 con 0x7f842c106570 2024-04-06T02:42:15.953 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.097+0000 7f84339d2700 1 -- 172.21.15.105:0/3597214779 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f842c19ae50 con 0x7f842c106570 2024-04-06T02:42:15.953 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.098+0000 7f84227fc700 1 -- 172.21.15.105:0/3597214779 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f842801a430 con 0x7f842c106570 2024-04-06T02:42:15.953 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.098+0000 7f84339d2700 1 -- 172.21.15.105:0/3597214779 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f842c0623c0 con 0x7f842c106570 2024-04-06T02:42:15.953 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.098+0000 7f84227fc700 1 --2- 172.21.15.105:0/3597214779 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f8418038330 0x7f841803a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:15.954 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.098+0000 7f84227fc700 1 -- 172.21.15.105:0/3597214779 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f842804c470 con 0x7f842c106570 2024-04-06T02:42:15.954 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.098+0000 7f8430f6d700 1 --2- 172.21.15.105:0/3597214779 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f8418038330 0x7f841803a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:15.954 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.098+0000 7f8430f6d700 1 --2- 172.21.15.105:0/3597214779 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f8418038330 0x7f841803a7f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f841c006fd0 tx=0x7f841c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:15.954 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.103+0000 7f84227fc700 1 -- 172.21.15.105:0/3597214779 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f842801a6e0 con 0x7f842c106570 2024-04-06T02:42:15.954 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.350+0000 7f84339d2700 1 -- 172.21.15.105:0/3597214779 --> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f842c103010 con 0x7f8418038330 2024-04-06T02:42:15.954 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.355+0000 7f84227fc700 1 -- 172.21.15.105:0/3597214779 <== mgr.14118 v2:172.21.15.105:6800/1394104199 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f842c103010 con 0x7f8418038330 2024-04-06T02:42:15.954 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.357+0000 7f84339d2700 1 -- 172.21.15.105:0/3597214779 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f8418038330 msgr2=0x7f841803a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:15.954 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.357+0000 7f84339d2700 1 --2- 172.21.15.105:0/3597214779 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f8418038330 0x7f841803a7f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f841c006fd0 tx=0x7f841c006e40 comp rx=0 tx=0).stop 2024-04-06T02:42:15.954 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.357+0000 7f84339d2700 1 -- 172.21.15.105:0/3597214779 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f842c106570 msgr2=0x7f842c19a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:15.954 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.357+0000 7f84339d2700 1 --2- 172.21.15.105:0/3597214779 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f842c106570 0x7f842c19a270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8428011670 tx=0x7f8428011aa0 comp rx=0 tx=0).stop 2024-04-06T02:42:15.954 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.357+0000 7f84339d2700 1 -- 172.21.15.105:0/3597214779 shutdown_connections 2024-04-06T02:42:15.954 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.357+0000 7f84339d2700 1 --2- 172.21.15.105:0/3597214779 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f8418038330 0x7f841803a7f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:15.954 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.357+0000 7f84339d2700 1 --2- 172.21.15.105:0/3597214779 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f842c106570 0x7f842c19a270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:15.954 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.357+0000 7f84339d2700 1 -- 172.21.15.105:0/3597214779 >> 172.21.15.105:0/3597214779 conn(0x7f842c101b10 msgr2=0x7f842c102900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:15.955 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.357+0000 7f84339d2700 1 -- 172.21.15.105:0/3597214779 shutdown_connections 2024-04-06T02:42:15.955 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:15.357+0000 7f84339d2700 1 -- 172.21.15.105:0/3597214779 wait complete. 2024-04-06T02:42:15.955 INFO:teuthology.orchestra.run.smithi105.stdout:Deploying crash service with default placement... 2024-04-06T02:42:16.423 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:16 smithi105 ceph-mon[29228]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:42:16.423 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:16 smithi105 ceph-mon[29228]: Saving service mgr spec with placement count:2 2024-04-06T02:42:16.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:16 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:17.652 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-04-06T02:42:17.652 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.839+0000 7f8d6a814700 1 Processor -- start 2024-04-06T02:42:17.652 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.839+0000 7f8d6a814700 1 -- start start 2024-04-06T02:42:17.652 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.839+0000 7f8d6a814700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d64107010 0x7f8d64109400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:17.652 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.839+0000 7f8d6a814700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d64079e50 con 0x7f8d64107010 2024-04-06T02:42:17.652 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.839+0000 7f8d63fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d64107010 0x7f8d64109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:17.652 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.840+0000 7f8d63fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d64107010 0x7f8d64109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50292/0 (socket says 172.21.15.105:50292) 2024-04-06T02:42:17.652 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.840+0000 7f8d63fff700 1 -- 172.21.15.105:0/1297814275 learned_addr learned my addr 172.21.15.105:0/1297814275 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:17.652 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.840+0000 7f8d63fff700 1 -- 172.21.15.105:0/1297814275 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d64109940 con 0x7f8d64107010 2024-04-06T02:42:17.652 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.840+0000 7f8d63fff700 1 --2- 172.21.15.105:0/1297814275 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d64107010 0x7f8d64109400 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8d540098a0 tx=0x7f8d54009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d426e78dbbc00327 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:17.652 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.840+0000 7f8d62ffd700 1 -- 172.21.15.105:0/1297814275 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d54004180 con 0x7f8d64107010 2024-04-06T02:42:17.653 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.840+0000 7f8d62ffd700 1 -- 172.21.15.105:0/1297814275 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8d540042e0 con 0x7f8d64107010 2024-04-06T02:42:17.653 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.840+0000 7f8d62ffd700 1 -- 172.21.15.105:0/1297814275 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d54011af0 con 0x7f8d64107010 2024-04-06T02:42:17.653 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.841+0000 7f8d6a814700 1 -- 172.21.15.105:0/1297814275 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d64107010 msgr2=0x7f8d64109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:17.653 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.841+0000 7f8d6a814700 1 --2- 172.21.15.105:0/1297814275 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d64107010 0x7f8d64109400 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8d540098a0 tx=0x7f8d54009bb0 comp rx=0 tx=0).stop 2024-04-06T02:42:17.653 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.841+0000 7f8d6a814700 1 -- 172.21.15.105:0/1297814275 shutdown_connections 2024-04-06T02:42:17.653 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.841+0000 7f8d6a814700 1 --2- 172.21.15.105:0/1297814275 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d64107010 0x7f8d64109400 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:17.653 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.841+0000 7f8d6a814700 1 -- 172.21.15.105:0/1297814275 >> 172.21.15.105:0/1297814275 conn(0x7f8d64100a20 msgr2=0x7f8d64102e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:17.653 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.841+0000 7f8d6a814700 1 -- 172.21.15.105:0/1297814275 shutdown_connections 2024-04-06T02:42:17.653 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.841+0000 7f8d6a814700 1 -- 172.21.15.105:0/1297814275 wait complete. 2024-04-06T02:42:17.653 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.841+0000 7f8d6a814700 1 Processor -- start 2024-04-06T02:42:17.653 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.841+0000 7f8d6a814700 1 -- start start 2024-04-06T02:42:17.653 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.842+0000 7f8d6a814700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d64107010 0x7f8d64106970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:17.653 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.842+0000 7f8d6a814700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d64106eb0 con 0x7f8d64107010 2024-04-06T02:42:17.654 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.842+0000 7f8d63fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d64107010 0x7f8d64106970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:17.654 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.842+0000 7f8d63fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d64107010 0x7f8d64106970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50302/0 (socket says 172.21.15.105:50302) 2024-04-06T02:42:17.654 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.842+0000 7f8d63fff700 1 -- 172.21.15.105:0/1931613901 learned_addr learned my addr 172.21.15.105:0/1931613901 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:17.654 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.842+0000 7f8d63fff700 1 -- 172.21.15.105:0/1931613901 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d54009550 con 0x7f8d64107010 2024-04-06T02:42:17.654 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.842+0000 7f8d63fff700 1 --2- 172.21.15.105:0/1931613901 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d64107010 0x7f8d64106970 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8d54011670 tx=0x7f8d54011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:17.654 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.842+0000 7f8d617fa700 1 -- 172.21.15.105:0/1931613901 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d54003a40 con 0x7f8d64107010 2024-04-06T02:42:17.654 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.842+0000 7f8d617fa700 1 -- 172.21.15.105:0/1931613901 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8d54003ba0 con 0x7f8d64107010 2024-04-06T02:42:17.654 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.842+0000 7f8d6a814700 1 -- 172.21.15.105:0/1931613901 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d64104fc0 con 0x7f8d64107010 2024-04-06T02:42:17.654 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.842+0000 7f8d617fa700 1 -- 172.21.15.105:0/1931613901 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d54024790 con 0x7f8d64107010 2024-04-06T02:42:17.654 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.842+0000 7f8d6a814700 1 -- 172.21.15.105:0/1931613901 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d64105460 con 0x7f8d64107010 2024-04-06T02:42:17.654 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.843+0000 7f8d617fa700 1 -- 172.21.15.105:0/1931613901 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f8d5402d3f0 con 0x7f8d64107010 2024-04-06T02:42:17.654 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.843+0000 7f8d617fa700 1 --2- 172.21.15.105:0/1931613901 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f8d4c038370 0x7f8d4c03a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:17.654 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.843+0000 7f8d6a814700 1 -- 172.21.15.105:0/1931613901 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d64062430 con 0x7f8d64107010 2024-04-06T02:42:17.655 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.843+0000 7f8d617fa700 1 -- 172.21.15.105:0/1931613901 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f8d5404c460 con 0x7f8d64107010 2024-04-06T02:42:17.655 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.844+0000 7f8d637fe700 1 --2- 172.21.15.105:0/1931613901 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f8d4c038370 0x7f8d4c03a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:17.655 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.844+0000 7f8d637fe700 1 --2- 172.21.15.105:0/1931613901 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f8d4c038370 0x7f8d4c03a830 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8d58006fd0 tx=0x7f8d58006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:17.655 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:16.848+0000 7f8d617fa700 1 -- 172.21.15.105:0/1931613901 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8d64062430 con 0x7f8d64107010 2024-04-06T02:42:17.655 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:17.057+0000 7f8d6a814700 1 -- 172.21.15.105:0/1931613901 --> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d64062430 con 0x7f8d4c038370 2024-04-06T02:42:17.655 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:17.062+0000 7f8d617fa700 1 -- 172.21.15.105:0/1931613901 <== mgr.14118 v2:172.21.15.105:6800/1394104199 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f8d64062430 con 0x7f8d4c038370 2024-04-06T02:42:17.655 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:17.064+0000 7f8d6a814700 1 -- 172.21.15.105:0/1931613901 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f8d4c038370 msgr2=0x7f8d4c03a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:17.655 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:17.064+0000 7f8d6a814700 1 --2- 172.21.15.105:0/1931613901 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f8d4c038370 0x7f8d4c03a830 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8d58006fd0 tx=0x7f8d58006e40 comp rx=0 tx=0).stop 2024-04-06T02:42:17.655 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:17.064+0000 7f8d6a814700 1 -- 172.21.15.105:0/1931613901 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d64107010 msgr2=0x7f8d64106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:17.655 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:17.064+0000 7f8d6a814700 1 --2- 172.21.15.105:0/1931613901 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d64107010 0x7f8d64106970 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8d54011670 tx=0x7f8d54011b50 comp rx=0 tx=0).stop 2024-04-06T02:42:17.655 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:17.064+0000 7f8d6a814700 1 -- 172.21.15.105:0/1931613901 shutdown_connections 2024-04-06T02:42:17.655 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:17.064+0000 7f8d6a814700 1 --2- 172.21.15.105:0/1931613901 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f8d4c038370 0x7f8d4c03a830 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:17.656 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:17.064+0000 7f8d6a814700 1 --2- 172.21.15.105:0/1931613901 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d64107010 0x7f8d64106970 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:17.656 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:17.064+0000 7f8d6a814700 1 -- 172.21.15.105:0/1931613901 >> 172.21.15.105:0/1931613901 conn(0x7f8d64100a20 msgr2=0x7f8d641017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:17.656 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:17.065+0000 7f8d6a814700 1 -- 172.21.15.105:0/1931613901 shutdown_connections 2024-04-06T02:42:17.656 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:17.065+0000 7f8d6a814700 1 -- 172.21.15.105:0/1931613901 wait complete. 2024-04-06T02:42:17.656 INFO:teuthology.orchestra.run.smithi105.stdout:Deploying ceph-exporter service with default placement... 2024-04-06T02:42:18.089 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:17 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:18.089 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:17 smithi105 ceph-mon[29228]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:42:18.089 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:17 smithi105 ceph-mon[29228]: Saving service crash spec with placement * 2024-04-06T02:42:18.089 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:17 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:18.850 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout Scheduled ceph-exporter update... 2024-04-06T02:42:18.850 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.120+0000 7f4a1894d700 1 Processor -- start 2024-04-06T02:42:18.850 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.120+0000 7f4a1894d700 1 -- start start 2024-04-06T02:42:18.850 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.121+0000 7f4a1894d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4a14105470 0x7f4a14105890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:18.851 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.121+0000 7f4a1894d700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a14105e60 con 0x7f4a14105470 2024-04-06T02:42:18.851 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.121+0000 7f4a1259c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4a14105470 0x7f4a14105890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:18.851 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.121+0000 7f4a1259c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4a14105470 0x7f4a14105890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50310/0 (socket says 172.21.15.105:50310) 2024-04-06T02:42:18.851 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.121+0000 7f4a1259c700 1 -- 172.21.15.105:0/2082990313 learned_addr learned my addr 172.21.15.105:0/2082990313 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:18.851 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.121+0000 7f4a1259c700 1 -- 172.21.15.105:0/2082990313 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a14106670 con 0x7f4a14105470 2024-04-06T02:42:18.851 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.122+0000 7f4a1259c700 1 --2- 172.21.15.105:0/2082990313 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4a14105470 0x7f4a14105890 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4a080098a0 tx=0x7f4a08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65850f01ef260cf5 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:18.851 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.122+0000 7f4a1159a700 1 -- 172.21.15.105:0/2082990313 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a08004180 con 0x7f4a14105470 2024-04-06T02:42:18.851 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.122+0000 7f4a1159a700 1 -- 172.21.15.105:0/2082990313 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4a080042e0 con 0x7f4a14105470 2024-04-06T02:42:18.851 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.122+0000 7f4a1159a700 1 -- 172.21.15.105:0/2082990313 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a08011af0 con 0x7f4a14105470 2024-04-06T02:42:18.851 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.122+0000 7f4a1894d700 1 -- 172.21.15.105:0/2082990313 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4a14105470 msgr2=0x7f4a14105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:18.851 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.122+0000 7f4a1894d700 1 --2- 172.21.15.105:0/2082990313 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4a14105470 0x7f4a14105890 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4a080098a0 tx=0x7f4a08009bb0 comp rx=0 tx=0).stop 2024-04-06T02:42:18.851 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.122+0000 7f4a1894d700 1 -- 172.21.15.105:0/2082990313 shutdown_connections 2024-04-06T02:42:18.851 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.122+0000 7f4a1894d700 1 --2- 172.21.15.105:0/2082990313 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4a14105470 0x7f4a14105890 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:18.851 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.122+0000 7f4a1894d700 1 -- 172.21.15.105:0/2082990313 >> 172.21.15.105:0/2082990313 conn(0x7f4a141009f0 msgr2=0x7f4a14102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:18.852 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.122+0000 7f4a1894d700 1 -- 172.21.15.105:0/2082990313 shutdown_connections 2024-04-06T02:42:18.852 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.122+0000 7f4a1894d700 1 -- 172.21.15.105:0/2082990313 wait complete. 2024-04-06T02:42:18.852 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.123+0000 7f4a1894d700 1 Processor -- start 2024-04-06T02:42:18.852 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.123+0000 7f4a1894d700 1 -- start start 2024-04-06T02:42:18.852 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.123+0000 7f4a1894d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4a14105470 0x7f4a14190660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:18.852 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.123+0000 7f4a1894d700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a14190ba0 con 0x7f4a14105470 2024-04-06T02:42:18.852 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.123+0000 7f4a1259c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4a14105470 0x7f4a14190660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:18.852 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.123+0000 7f4a1259c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4a14105470 0x7f4a14190660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50318/0 (socket says 172.21.15.105:50318) 2024-04-06T02:42:18.852 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.123+0000 7f4a1259c700 1 -- 172.21.15.105:0/2779404024 learned_addr learned my addr 172.21.15.105:0/2779404024 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:18.852 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.124+0000 7f4a1259c700 1 -- 172.21.15.105:0/2779404024 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a08009550 con 0x7f4a14105470 2024-04-06T02:42:18.852 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.124+0000 7f4a1259c700 1 --2- 172.21.15.105:0/2779404024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4a14105470 0x7f4a14190660 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4a08006ba0 tx=0x7f4a08011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:18.852 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.124+0000 7f4a037fe700 1 -- 172.21.15.105:0/2779404024 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a08003b00 con 0x7f4a14105470 2024-04-06T02:42:18.852 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.124+0000 7f4a037fe700 1 -- 172.21.15.105:0/2779404024 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4a0801a8e0 con 0x7f4a14105470 2024-04-06T02:42:18.852 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.124+0000 7f4a1894d700 1 -- 172.21.15.105:0/2779404024 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a1418ecb0 con 0x7f4a14105470 2024-04-06T02:42:18.853 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.124+0000 7f4a037fe700 1 -- 172.21.15.105:0/2779404024 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a08019990 con 0x7f4a14105470 2024-04-06T02:42:18.853 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.124+0000 7f4a1894d700 1 -- 172.21.15.105:0/2779404024 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a1418f150 con 0x7f4a14105470 2024-04-06T02:42:18.853 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.125+0000 7f4a037fe700 1 -- 172.21.15.105:0/2779404024 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f4a08019c00 con 0x7f4a14105470 2024-04-06T02:42:18.853 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.125+0000 7f4a1894d700 1 -- 172.21.15.105:0/2779404024 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a140623c0 con 0x7f4a14105470 2024-04-06T02:42:18.853 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.125+0000 7f4a037fe700 1 --2- 172.21.15.105:0/2779404024 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f49fc038320 0x7f49fc03a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:18.853 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.125+0000 7f4a037fe700 1 -- 172.21.15.105:0/2779404024 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4a0804c590 con 0x7f4a14105470 2024-04-06T02:42:18.853 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.125+0000 7f4a11d9b700 1 --2- 172.21.15.105:0/2779404024 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f49fc038320 0x7f49fc03a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:18.853 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.126+0000 7f4a11d9b700 1 --2- 172.21.15.105:0/2779404024 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f49fc038320 0x7f49fc03a7e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4a04006fd0 tx=0x7f4a04006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:18.853 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.130+0000 7f4a037fe700 1 -- 172.21.15.105:0/2779404024 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4a0802d440 con 0x7f4a14105470 2024-04-06T02:42:18.853 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.339+0000 7f4a1894d700 1 -- 172.21.15.105:0/2779404024 --> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a14102b40 con 0x7f49fc038320 2024-04-06T02:42:18.853 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.345+0000 7f4a037fe700 1 -- 172.21.15.105:0/2779404024 <== mgr.14118 v2:172.21.15.105:6800/1394104199 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f4a14102b40 con 0x7f49fc038320 2024-04-06T02:42:18.853 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.347+0000 7f4a1894d700 1 -- 172.21.15.105:0/2779404024 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f49fc038320 msgr2=0x7f49fc03a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:18.853 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.347+0000 7f4a1894d700 1 --2- 172.21.15.105:0/2779404024 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f49fc038320 0x7f49fc03a7e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4a04006fd0 tx=0x7f4a04006e40 comp rx=0 tx=0).stop 2024-04-06T02:42:18.854 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.347+0000 7f4a1894d700 1 -- 172.21.15.105:0/2779404024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4a14105470 msgr2=0x7f4a14190660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:18.854 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.347+0000 7f4a1894d700 1 --2- 172.21.15.105:0/2779404024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4a14105470 0x7f4a14190660 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4a08006ba0 tx=0x7f4a08011b50 comp rx=0 tx=0).stop 2024-04-06T02:42:18.854 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.347+0000 7f4a1894d700 1 -- 172.21.15.105:0/2779404024 shutdown_connections 2024-04-06T02:42:18.854 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.347+0000 7f4a1894d700 1 --2- 172.21.15.105:0/2779404024 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f49fc038320 0x7f49fc03a7e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:18.854 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.347+0000 7f4a1894d700 1 --2- 172.21.15.105:0/2779404024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4a14105470 0x7f4a14190660 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:18.854 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.347+0000 7f4a1894d700 1 -- 172.21.15.105:0/2779404024 >> 172.21.15.105:0/2779404024 conn(0x7f4a141009f0 msgr2=0x7f4a140740f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:18.854 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.348+0000 7f4a1894d700 1 -- 172.21.15.105:0/2779404024 shutdown_connections 2024-04-06T02:42:18.854 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:18.348+0000 7f4a1894d700 1 -- 172.21.15.105:0/2779404024 wait complete. 2024-04-06T02:42:18.854 INFO:teuthology.orchestra.run.smithi105.stdout:Deploying prometheus service with default placement... 2024-04-06T02:42:19.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:19 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:19.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:19 smithi105 ceph-mon[29228]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:42:19.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:19 smithi105 ceph-mon[29228]: Saving service ceph-exporter spec with placement * 2024-04-06T02:42:19.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:19 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:20.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-04-06T02:42:20.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.317+0000 7f6dec960700 1 Processor -- start 2024-04-06T02:42:20.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.317+0000 7f6dec960700 1 -- start start 2024-04-06T02:42:20.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.318+0000 7f6dec960700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6de81070c0 0x7f6de81094b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:20.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.318+0000 7f6dec960700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de80745a0 con 0x7f6de81070c0 2024-04-06T02:42:20.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.318+0000 7f6de659c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6de81070c0 0x7f6de81094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:20.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.318+0000 7f6de659c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6de81070c0 0x7f6de81094b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50332/0 (socket says 172.21.15.105:50332) 2024-04-06T02:42:20.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.318+0000 7f6de659c700 1 -- 172.21.15.105:0/4180588606 learned_addr learned my addr 172.21.15.105:0/4180588606 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:20.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.318+0000 7f6de659c700 1 -- 172.21.15.105:0/4180588606 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de80746e0 con 0x7f6de81070c0 2024-04-06T02:42:20.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.318+0000 7f6de659c700 1 --2- 172.21.15.105:0/4180588606 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6de81070c0 0x7f6de81094b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6dd80098a0 tx=0x7f6dd8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ab924403715d2d server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:20.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.318+0000 7f6de559a700 1 -- 172.21.15.105:0/4180588606 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6dd8004180 con 0x7f6de81070c0 2024-04-06T02:42:20.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.318+0000 7f6de559a700 1 -- 172.21.15.105:0/4180588606 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6dd80042e0 con 0x7f6de81070c0 2024-04-06T02:42:20.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.318+0000 7f6de559a700 1 -- 172.21.15.105:0/4180588606 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6dd8011af0 con 0x7f6de81070c0 2024-04-06T02:42:20.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.319+0000 7f6dec960700 1 -- 172.21.15.105:0/4180588606 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6de81070c0 msgr2=0x7f6de81094b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:20.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.319+0000 7f6dec960700 1 --2- 172.21.15.105:0/4180588606 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6de81070c0 0x7f6de81094b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6dd80098a0 tx=0x7f6dd8009bb0 comp rx=0 tx=0).stop 2024-04-06T02:42:20.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.319+0000 7f6dec960700 1 -- 172.21.15.105:0/4180588606 shutdown_connections 2024-04-06T02:42:20.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.319+0000 7f6dec960700 1 --2- 172.21.15.105:0/4180588606 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6de81070c0 0x7f6de81094b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:20.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.319+0000 7f6dec960700 1 -- 172.21.15.105:0/4180588606 >> 172.21.15.105:0/4180588606 conn(0x7f6de81009f0 msgr2=0x7f6de8102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:20.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.319+0000 7f6dec960700 1 -- 172.21.15.105:0/4180588606 shutdown_connections 2024-04-06T02:42:20.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.319+0000 7f6dec960700 1 -- 172.21.15.105:0/4180588606 wait complete. 2024-04-06T02:42:20.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.320+0000 7f6dec960700 1 Processor -- start 2024-04-06T02:42:20.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.320+0000 7f6dec960700 1 -- start start 2024-04-06T02:42:20.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.320+0000 7f6dec960700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6de81070c0 0x7f6de819a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:20.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.320+0000 7f6dec960700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de819a6e0 con 0x7f6de81070c0 2024-04-06T02:42:20.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.320+0000 7f6de659c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6de81070c0 0x7f6de819a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:20.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.320+0000 7f6de659c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6de81070c0 0x7f6de819a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50338/0 (socket says 172.21.15.105:50338) 2024-04-06T02:42:20.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.320+0000 7f6de659c700 1 -- 172.21.15.105:0/574312267 learned_addr learned my addr 172.21.15.105:0/574312267 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:20.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.320+0000 7f6de659c700 1 -- 172.21.15.105:0/574312267 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd8009550 con 0x7f6de81070c0 2024-04-06T02:42:20.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.320+0000 7f6de659c700 1 --2- 172.21.15.105:0/574312267 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6de81070c0 0x7f6de819a1a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8011ad0 tx=0x7f6dd8011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:20.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.320+0000 7f6dd77fe700 1 -- 172.21.15.105:0/574312267 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6dd8003ab0 con 0x7f6de81070c0 2024-04-06T02:42:20.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.320+0000 7f6dd77fe700 1 -- 172.21.15.105:0/574312267 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6dd8003c10 con 0x7f6de81070c0 2024-04-06T02:42:20.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.320+0000 7f6dec960700 1 -- 172.21.15.105:0/574312267 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de819a8e0 con 0x7f6de81070c0 2024-04-06T02:42:20.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.320+0000 7f6dd77fe700 1 -- 172.21.15.105:0/574312267 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6dd80198e0 con 0x7f6de81070c0 2024-04-06T02:42:20.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.320+0000 7f6dec960700 1 -- 172.21.15.105:0/574312267 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de819ad80 con 0x7f6de81070c0 2024-04-06T02:42:20.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.321+0000 7f6dd77fe700 1 -- 172.21.15.105:0/574312267 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f6dd8019ba0 con 0x7f6de81070c0 2024-04-06T02:42:20.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.321+0000 7f6dd77fe700 1 --2- 172.21.15.105:0/574312267 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f6dd0038320 0x7f6dd003a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:20.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.321+0000 7f6dec960700 1 -- 172.21.15.105:0/574312267 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de810a710 con 0x7f6de81070c0 2024-04-06T02:42:20.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.321+0000 7f6dd77fe700 1 -- 172.21.15.105:0/574312267 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6dd804c410 con 0x7f6de81070c0 2024-04-06T02:42:20.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.321+0000 7f6de5d9b700 1 --2- 172.21.15.105:0/574312267 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f6dd0038320 0x7f6dd003a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:20.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.322+0000 7f6de5d9b700 1 --2- 172.21.15.105:0/574312267 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f6dd0038320 0x7f6dd003a7e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc006fd0 tx=0x7f6ddc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:20.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.326+0000 7f6dd77fe700 1 -- 172.21.15.105:0/574312267 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6dd802b300 con 0x7f6de81070c0 2024-04-06T02:42:20.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.533+0000 7f6dec960700 1 -- 172.21.15.105:0/574312267 --> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f6de8101e50 con 0x7f6dd0038320 2024-04-06T02:42:20.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.538+0000 7f6dd77fe700 1 -- 172.21.15.105:0/574312267 <== mgr.14118 v2:172.21.15.105:6800/1394104199 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f6de8101e50 con 0x7f6dd0038320 2024-04-06T02:42:20.062 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.540+0000 7f6dec960700 1 -- 172.21.15.105:0/574312267 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f6dd0038320 msgr2=0x7f6dd003a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:20.062 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.540+0000 7f6dec960700 1 --2- 172.21.15.105:0/574312267 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f6dd0038320 0x7f6dd003a7e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc006fd0 tx=0x7f6ddc006e40 comp rx=0 tx=0).stop 2024-04-06T02:42:20.062 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.540+0000 7f6dec960700 1 -- 172.21.15.105:0/574312267 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6de81070c0 msgr2=0x7f6de819a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:20.062 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.540+0000 7f6dec960700 1 --2- 172.21.15.105:0/574312267 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6de81070c0 0x7f6de819a1a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8011ad0 tx=0x7f6dd8011bb0 comp rx=0 tx=0).stop 2024-04-06T02:42:20.062 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.540+0000 7f6dec960700 1 -- 172.21.15.105:0/574312267 shutdown_connections 2024-04-06T02:42:20.062 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.540+0000 7f6dec960700 1 --2- 172.21.15.105:0/574312267 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f6dd0038320 0x7f6dd003a7e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:20.062 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.540+0000 7f6dec960700 1 --2- 172.21.15.105:0/574312267 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6de81070c0 0x7f6de819a1a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:20.062 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.540+0000 7f6dec960700 1 -- 172.21.15.105:0/574312267 >> 172.21.15.105:0/574312267 conn(0x7f6de81009f0 msgr2=0x7f6de8101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:20.062 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.540+0000 7f6dec960700 1 -- 172.21.15.105:0/574312267 shutdown_connections 2024-04-06T02:42:20.062 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:19.540+0000 7f6dec960700 1 -- 172.21.15.105:0/574312267 wait complete. 2024-04-06T02:42:20.062 INFO:teuthology.orchestra.run.smithi105.stdout:Deploying grafana service with default placement... 2024-04-06T02:42:20.822 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:20 smithi105 ceph-mon[29228]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:42:20.822 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:20 smithi105 ceph-mon[29228]: Saving service prometheus spec with placement count:1 2024-04-06T02:42:20.823 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:20 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:21.836 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-04-06T02:42:21.836 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.937+0000 7f9e9f378700 1 Processor -- start 2024-04-06T02:42:21.836 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.937+0000 7f9e9f378700 1 -- start start 2024-04-06T02:42:21.836 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.937+0000 7f9e9f378700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e98106580 0x7f9e981069a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:21.836 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.937+0000 7f9e9f378700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e98106f70 con 0x7f9e98106580 2024-04-06T02:42:21.836 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.937+0000 7f9e9d114700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e98106580 0x7f9e981069a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:21.836 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.937+0000 7f9e9d114700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e98106580 0x7f9e981069a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53154/0 (socket says 172.21.15.105:53154) 2024-04-06T02:42:21.836 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.937+0000 7f9e9d114700 1 -- 172.21.15.105:0/2830351401 learned_addr learned my addr 172.21.15.105:0/2830351401 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:21.836 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.937+0000 7f9e9d114700 1 -- 172.21.15.105:0/2830351401 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e98107780 con 0x7f9e98106580 2024-04-06T02:42:21.836 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.938+0000 7f9e9d114700 1 --2- 172.21.15.105:0/2830351401 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e98106580 0x7f9e981069a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9e880098a0 tx=0x7f9e88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8480a4842192ebd server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:21.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.938+0000 7f9e8ffff700 1 -- 172.21.15.105:0/2830351401 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e88004180 con 0x7f9e98106580 2024-04-06T02:42:21.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.938+0000 7f9e8ffff700 1 -- 172.21.15.105:0/2830351401 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9e880042e0 con 0x7f9e98106580 2024-04-06T02:42:21.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.938+0000 7f9e8ffff700 1 -- 172.21.15.105:0/2830351401 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e88011af0 con 0x7f9e98106580 2024-04-06T02:42:21.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.938+0000 7f9e9f378700 1 -- 172.21.15.105:0/2830351401 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e98106580 msgr2=0x7f9e981069a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:21.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.938+0000 7f9e9f378700 1 --2- 172.21.15.105:0/2830351401 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e98106580 0x7f9e981069a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9e880098a0 tx=0x7f9e88009bb0 comp rx=0 tx=0).stop 2024-04-06T02:42:21.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.938+0000 7f9e9f378700 1 -- 172.21.15.105:0/2830351401 shutdown_connections 2024-04-06T02:42:21.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.938+0000 7f9e9f378700 1 --2- 172.21.15.105:0/2830351401 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e98106580 0x7f9e981069a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:21.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.938+0000 7f9e9f378700 1 -- 172.21.15.105:0/2830351401 >> 172.21.15.105:0/2830351401 conn(0x7f9e98101b00 msgr2=0x7f9e98103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:21.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.938+0000 7f9e9f378700 1 -- 172.21.15.105:0/2830351401 shutdown_connections 2024-04-06T02:42:21.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.938+0000 7f9e9f378700 1 -- 172.21.15.105:0/2830351401 wait complete. 2024-04-06T02:42:21.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.939+0000 7f9e9f378700 1 Processor -- start 2024-04-06T02:42:21.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.939+0000 7f9e9f378700 1 -- start start 2024-04-06T02:42:21.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.939+0000 7f9e9f378700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e98106580 0x7f9e9819a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:21.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.939+0000 7f9e9f378700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9819a7d0 con 0x7f9e98106580 2024-04-06T02:42:21.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.939+0000 7f9e9d114700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e98106580 0x7f9e9819a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:21.838 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.939+0000 7f9e9d114700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e98106580 0x7f9e9819a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53170/0 (socket says 172.21.15.105:53170) 2024-04-06T02:42:21.838 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.939+0000 7f9e9d114700 1 -- 172.21.15.105:0/2774885973 learned_addr learned my addr 172.21.15.105:0/2774885973 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:21.838 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.940+0000 7f9e9d114700 1 -- 172.21.15.105:0/2774885973 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e88009550 con 0x7f9e98106580 2024-04-06T02:42:21.838 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.940+0000 7f9e9d114700 1 --2- 172.21.15.105:0/2774885973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e98106580 0x7f9e9819a290 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9e88011670 tx=0x7f9e88011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:21.838 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.940+0000 7f9e8e7fc700 1 -- 172.21.15.105:0/2774885973 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e880039c0 con 0x7f9e98106580 2024-04-06T02:42:21.838 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.940+0000 7f9e8e7fc700 1 -- 172.21.15.105:0/2774885973 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9e88003b20 con 0x7f9e98106580 2024-04-06T02:42:21.838 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.940+0000 7f9e9f378700 1 -- 172.21.15.105:0/2774885973 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e9819a9d0 con 0x7f9e98106580 2024-04-06T02:42:21.838 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.940+0000 7f9e8e7fc700 1 -- 172.21.15.105:0/2774885973 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e880247f0 con 0x7f9e98106580 2024-04-06T02:42:21.838 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.940+0000 7f9e9f378700 1 -- 172.21.15.105:0/2774885973 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e9819ae70 con 0x7f9e98106580 2024-04-06T02:42:21.838 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.941+0000 7f9e8e7fc700 1 -- 172.21.15.105:0/2774885973 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f9e88024a60 con 0x7f9e98106580 2024-04-06T02:42:21.838 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.941+0000 7f9e8e7fc700 1 --2- 172.21.15.105:0/2774885973 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f9e84038330 0x7f9e8403a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:21.838 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.941+0000 7f9e8e7fc700 1 -- 172.21.15.105:0/2774885973 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f9e8804c280 con 0x7f9e98106580 2024-04-06T02:42:21.838 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.941+0000 7f9e9f378700 1 -- 172.21.15.105:0/2774885973 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e980623c0 con 0x7f9e98106580 2024-04-06T02:42:21.838 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.941+0000 7f9e9c913700 1 --2- 172.21.15.105:0/2774885973 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f9e84038330 0x7f9e8403a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:21.838 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.941+0000 7f9e9c913700 1 --2- 172.21.15.105:0/2774885973 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f9e84038330 0x7f9e8403a7f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9e94006fd0 tx=0x7f9e94006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:21.839 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:20.946+0000 7f9e8e7fc700 1 -- 172.21.15.105:0/2774885973 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f9e88024d10 con 0x7f9e98106580 2024-04-06T02:42:21.839 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:21.152+0000 7f9e9f378700 1 -- 172.21.15.105:0/2774885973 --> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e9810a980 con 0x7f9e84038330 2024-04-06T02:42:21.839 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:21.157+0000 7f9e8e7fc700 1 -- 172.21.15.105:0/2774885973 <== mgr.14118 v2:172.21.15.105:6800/1394104199 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f9e9810a980 con 0x7f9e84038330 2024-04-06T02:42:21.839 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:21.159+0000 7f9e9f378700 1 -- 172.21.15.105:0/2774885973 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f9e84038330 msgr2=0x7f9e8403a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:21.839 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:21.159+0000 7f9e9f378700 1 --2- 172.21.15.105:0/2774885973 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f9e84038330 0x7f9e8403a7f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9e94006fd0 tx=0x7f9e94006e40 comp rx=0 tx=0).stop 2024-04-06T02:42:21.839 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:21.159+0000 7f9e9f378700 1 -- 172.21.15.105:0/2774885973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e98106580 msgr2=0x7f9e9819a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:21.839 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:21.159+0000 7f9e9f378700 1 --2- 172.21.15.105:0/2774885973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e98106580 0x7f9e9819a290 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9e88011670 tx=0x7f9e88011aa0 comp rx=0 tx=0).stop 2024-04-06T02:42:21.839 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:21.159+0000 7f9e9f378700 1 -- 172.21.15.105:0/2774885973 shutdown_connections 2024-04-06T02:42:21.839 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:21.159+0000 7f9e9f378700 1 --2- 172.21.15.105:0/2774885973 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f9e84038330 0x7f9e8403a7f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:21.839 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:21.159+0000 7f9e9f378700 1 --2- 172.21.15.105:0/2774885973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e98106580 0x7f9e9819a290 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:21.839 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:21.159+0000 7f9e9f378700 1 -- 172.21.15.105:0/2774885973 >> 172.21.15.105:0/2774885973 conn(0x7f9e98101b00 msgr2=0x7f9e981031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:21.839 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:21.159+0000 7f9e9f378700 1 -- 172.21.15.105:0/2774885973 shutdown_connections 2024-04-06T02:42:21.839 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:21.159+0000 7f9e9f378700 1 -- 172.21.15.105:0/2774885973 wait complete. 2024-04-06T02:42:21.839 INFO:teuthology.orchestra.run.smithi105.stdout:Deploying node-exporter service with default placement... 2024-04-06T02:42:22.256 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:22 smithi105 ceph-mon[29228]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:42:22.256 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:22 smithi105 ceph-mon[29228]: Saving service grafana spec with placement count:1 2024-04-06T02:42:22.256 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:22 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:23.327 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-04-06T02:42:23.327 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.282+0000 7fba75bef700 1 Processor -- start 2024-04-06T02:42:23.327 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.282+0000 7fba75bef700 1 -- start start 2024-04-06T02:42:23.327 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.282+0000 7fba75bef700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fba701069e0 0x7fba70106e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:23.327 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.282+0000 7fba75bef700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba701073d0 con 0x7fba701069e0 2024-04-06T02:42:23.327 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.282+0000 7fba6f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fba701069e0 0x7fba70106e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:23.327 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.282+0000 7fba6f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fba701069e0 0x7fba70106e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53184/0 (socket says 172.21.15.105:53184) 2024-04-06T02:42:23.327 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.282+0000 7fba6f7fe700 1 -- 172.21.15.105:0/2870217491 learned_addr learned my addr 172.21.15.105:0/2870217491 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:23.327 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.283+0000 7fba6f7fe700 1 -- 172.21.15.105:0/2870217491 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba70107510 con 0x7fba701069e0 2024-04-06T02:42:23.327 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.283+0000 7fba6f7fe700 1 --2- 172.21.15.105:0/2870217491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fba701069e0 0x7fba70106e00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fba60009e80 tx=0x7fba6000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34abf6bcb8f8c2f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:23.328 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.283+0000 7fba6e7fc700 1 -- 172.21.15.105:0/2870217491 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba6000e070 con 0x7fba701069e0 2024-04-06T02:42:23.328 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.283+0000 7fba6e7fc700 1 -- 172.21.15.105:0/2870217491 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fba6000b8c0 con 0x7fba701069e0 2024-04-06T02:42:23.328 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.283+0000 7fba6e7fc700 1 -- 172.21.15.105:0/2870217491 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba600044d0 con 0x7fba701069e0 2024-04-06T02:42:23.328 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.283+0000 7fba75bef700 1 -- 172.21.15.105:0/2870217491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fba701069e0 msgr2=0x7fba70106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:23.328 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.283+0000 7fba75bef700 1 --2- 172.21.15.105:0/2870217491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fba701069e0 0x7fba70106e00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fba60009e80 tx=0x7fba6000b330 comp rx=0 tx=0).stop 2024-04-06T02:42:23.328 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.283+0000 7fba75bef700 1 -- 172.21.15.105:0/2870217491 shutdown_connections 2024-04-06T02:42:23.328 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.283+0000 7fba75bef700 1 --2- 172.21.15.105:0/2870217491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fba701069e0 0x7fba70106e00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:23.328 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.283+0000 7fba75bef700 1 -- 172.21.15.105:0/2870217491 >> 172.21.15.105:0/2870217491 conn(0x7fba70100170 msgr2=0x7fba701025b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:23.328 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.284+0000 7fba75bef700 1 -- 172.21.15.105:0/2870217491 shutdown_connections 2024-04-06T02:42:23.328 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.284+0000 7fba75bef700 1 -- 172.21.15.105:0/2870217491 wait complete. 2024-04-06T02:42:23.328 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.284+0000 7fba75bef700 1 Processor -- start 2024-04-06T02:42:23.328 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.284+0000 7fba75bef700 1 -- start start 2024-04-06T02:42:23.328 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.284+0000 7fba75bef700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fba701069e0 0x7fba70105190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:23.328 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.284+0000 7fba75bef700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7019cf50 con 0x7fba701069e0 2024-04-06T02:42:23.329 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.284+0000 7fba6f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fba701069e0 0x7fba70105190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:23.329 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.284+0000 7fba6f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fba701069e0 0x7fba70105190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53192/0 (socket says 172.21.15.105:53192) 2024-04-06T02:42:23.329 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.284+0000 7fba6f7fe700 1 -- 172.21.15.105:0/689532784 learned_addr learned my addr 172.21.15.105:0/689532784 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:23.329 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.285+0000 7fba6f7fe700 1 -- 172.21.15.105:0/689532784 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba60009b30 con 0x7fba701069e0 2024-04-06T02:42:23.329 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.285+0000 7fba6f7fe700 1 --2- 172.21.15.105:0/689532784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fba701069e0 0x7fba70105190 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fba60004940 tx=0x7fba60004a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:23.329 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.285+0000 7fba6cff9700 1 -- 172.21.15.105:0/689532784 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba6000e040 con 0x7fba701069e0 2024-04-06T02:42:23.329 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.285+0000 7fba6cff9700 1 -- 172.21.15.105:0/689532784 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fba6000bc40 con 0x7fba701069e0 2024-04-06T02:42:23.329 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.285+0000 7fba75bef700 1 -- 172.21.15.105:0/689532784 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba7019d090 con 0x7fba701069e0 2024-04-06T02:42:23.329 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.285+0000 7fba6cff9700 1 -- 172.21.15.105:0/689532784 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba60012670 con 0x7fba701069e0 2024-04-06T02:42:23.329 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.285+0000 7fba75bef700 1 -- 172.21.15.105:0/689532784 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba7019d590 con 0x7fba701069e0 2024-04-06T02:42:23.329 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.286+0000 7fba6cff9700 1 -- 172.21.15.105:0/689532784 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fba60019040 con 0x7fba701069e0 2024-04-06T02:42:23.329 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.286+0000 7fba75bef700 1 -- 172.21.15.105:0/689532784 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba700623c0 con 0x7fba701069e0 2024-04-06T02:42:23.329 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.286+0000 7fba6cff9700 1 --2- 172.21.15.105:0/689532784 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fba58038320 0x7fba5803a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:23.329 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.286+0000 7fba6cff9700 1 -- 172.21.15.105:0/689532784 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fba6004bbf0 con 0x7fba701069e0 2024-04-06T02:42:23.330 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.286+0000 7fba6effd700 1 --2- 172.21.15.105:0/689532784 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fba58038320 0x7fba5803a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:23.330 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.287+0000 7fba6effd700 1 --2- 172.21.15.105:0/689532784 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fba58038320 0x7fba5803a7e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fba64006fd0 tx=0x7fba64006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:23.330 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.291+0000 7fba6cff9700 1 -- 172.21.15.105:0/689532784 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fba6001b4d0 con 0x7fba701069e0 2024-04-06T02:42:23.330 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.490+0000 7fba75bef700 1 -- 172.21.15.105:0/689532784 --> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fba70101640 con 0x7fba58038320 2024-04-06T02:42:23.330 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.497+0000 7fba6cff9700 1 -- 172.21.15.105:0/689532784 <== mgr.14118 v2:172.21.15.105:6800/1394104199 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fba70101640 con 0x7fba58038320 2024-04-06T02:42:23.330 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.499+0000 7fba75bef700 1 -- 172.21.15.105:0/689532784 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fba58038320 msgr2=0x7fba5803a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:23.330 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.499+0000 7fba75bef700 1 --2- 172.21.15.105:0/689532784 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fba58038320 0x7fba5803a7e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fba64006fd0 tx=0x7fba64006e40 comp rx=0 tx=0).stop 2024-04-06T02:42:23.330 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.499+0000 7fba75bef700 1 -- 172.21.15.105:0/689532784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fba701069e0 msgr2=0x7fba70105190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:23.330 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.499+0000 7fba75bef700 1 --2- 172.21.15.105:0/689532784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fba701069e0 0x7fba70105190 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fba60004940 tx=0x7fba60004a20 comp rx=0 tx=0).stop 2024-04-06T02:42:23.330 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.499+0000 7fba75bef700 1 -- 172.21.15.105:0/689532784 shutdown_connections 2024-04-06T02:42:23.330 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.499+0000 7fba75bef700 1 --2- 172.21.15.105:0/689532784 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fba58038320 0x7fba5803a7e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:23.330 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.499+0000 7fba75bef700 1 --2- 172.21.15.105:0/689532784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fba701069e0 0x7fba70105190 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:23.330 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.499+0000 7fba75bef700 1 -- 172.21.15.105:0/689532784 >> 172.21.15.105:0/689532784 conn(0x7fba70100170 msgr2=0x7fba70100f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:23.331 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.499+0000 7fba75bef700 1 -- 172.21.15.105:0/689532784 shutdown_connections 2024-04-06T02:42:23.331 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:22.499+0000 7fba75bef700 1 -- 172.21.15.105:0/689532784 wait complete. 2024-04-06T02:42:23.331 INFO:teuthology.orchestra.run.smithi105.stdout:Deploying alertmanager service with default placement... 2024-04-06T02:42:23.621 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:23 smithi105 ceph-mon[29228]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:42:23.621 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:23 smithi105 ceph-mon[29228]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:42:23.621 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:23 smithi105 ceph-mon[29228]: Saving service node-exporter spec with placement * 2024-04-06T02:42:23.621 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:23 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:24.541 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-04-06T02:42:24.542 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.738+0000 7faf35bbf700 1 Processor -- start 2024-04-06T02:42:24.542 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.738+0000 7faf35bbf700 1 -- start start 2024-04-06T02:42:24.542 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.738+0000 7faf35bbf700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf30105470 0x7faf30105890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:24.542 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.738+0000 7faf35bbf700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf30105e60 con 0x7faf30105470 2024-04-06T02:42:24.542 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.738+0000 7faf2f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf30105470 0x7faf30105890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:24.542 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.738+0000 7faf2f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf30105470 0x7faf30105890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53198/0 (socket says 172.21.15.105:53198) 2024-04-06T02:42:24.542 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.738+0000 7faf2f7fe700 1 -- 172.21.15.105:0/2965817056 learned_addr learned my addr 172.21.15.105:0/2965817056 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:24.542 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.739+0000 7faf2f7fe700 1 -- 172.21.15.105:0/2965817056 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf30106670 con 0x7faf30105470 2024-04-06T02:42:24.542 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.739+0000 7faf2f7fe700 1 --2- 172.21.15.105:0/2965817056 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf30105470 0x7faf30105890 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7faf240098a0 tx=0x7faf24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8980cdbf28868ecc server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:24.542 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.739+0000 7faf2e7fc700 1 -- 172.21.15.105:0/2965817056 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf24004180 con 0x7faf30105470 2024-04-06T02:42:24.542 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.739+0000 7faf2e7fc700 1 -- 172.21.15.105:0/2965817056 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7faf240042e0 con 0x7faf30105470 2024-04-06T02:42:24.542 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.739+0000 7faf2e7fc700 1 -- 172.21.15.105:0/2965817056 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf24011af0 con 0x7faf30105470 2024-04-06T02:42:24.543 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.740+0000 7faf35bbf700 1 -- 172.21.15.105:0/2965817056 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf30105470 msgr2=0x7faf30105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:24.543 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.740+0000 7faf35bbf700 1 --2- 172.21.15.105:0/2965817056 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf30105470 0x7faf30105890 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7faf240098a0 tx=0x7faf24009bb0 comp rx=0 tx=0).stop 2024-04-06T02:42:24.543 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.740+0000 7faf35bbf700 1 -- 172.21.15.105:0/2965817056 shutdown_connections 2024-04-06T02:42:24.543 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.740+0000 7faf35bbf700 1 --2- 172.21.15.105:0/2965817056 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf30105470 0x7faf30105890 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:24.543 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.740+0000 7faf35bbf700 1 -- 172.21.15.105:0/2965817056 >> 172.21.15.105:0/2965817056 conn(0x7faf301009f0 msgr2=0x7faf30102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:24.543 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.740+0000 7faf35bbf700 1 -- 172.21.15.105:0/2965817056 shutdown_connections 2024-04-06T02:42:24.543 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.740+0000 7faf35bbf700 1 -- 172.21.15.105:0/2965817056 wait complete. 2024-04-06T02:42:24.543 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.740+0000 7faf35bbf700 1 Processor -- start 2024-04-06T02:42:24.543 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.741+0000 7faf35bbf700 1 -- start start 2024-04-06T02:42:24.543 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.741+0000 7faf35bbf700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf30105470 0x7faf30197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:24.543 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.741+0000 7faf35bbf700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf30198520 con 0x7faf30105470 2024-04-06T02:42:24.543 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.741+0000 7faf2f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf30105470 0x7faf30197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:24.543 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.741+0000 7faf2f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf30105470 0x7faf30197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53200/0 (socket says 172.21.15.105:53200) 2024-04-06T02:42:24.543 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.741+0000 7faf2f7fe700 1 -- 172.21.15.105:0/1367459555 learned_addr learned my addr 172.21.15.105:0/1367459555 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:24.543 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.741+0000 7faf2f7fe700 1 -- 172.21.15.105:0/1367459555 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf24009550 con 0x7faf30105470 2024-04-06T02:42:24.544 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.741+0000 7faf2f7fe700 1 --2- 172.21.15.105:0/1367459555 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf30105470 0x7faf30197fe0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7faf24006ba0 tx=0x7faf24011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:24.544 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.741+0000 7faf2cff9700 1 -- 172.21.15.105:0/1367459555 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf24003b00 con 0x7faf30105470 2024-04-06T02:42:24.544 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.741+0000 7faf2cff9700 1 -- 172.21.15.105:0/1367459555 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7faf2401a8e0 con 0x7faf30105470 2024-04-06T02:42:24.544 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.741+0000 7faf2cff9700 1 -- 172.21.15.105:0/1367459555 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf24019990 con 0x7faf30105470 2024-04-06T02:42:24.544 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.742+0000 7faf35bbf700 1 -- 172.21.15.105:0/1367459555 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf30198720 con 0x7faf30105470 2024-04-06T02:42:24.544 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.742+0000 7faf35bbf700 1 -- 172.21.15.105:0/1367459555 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf30198bc0 con 0x7faf30105470 2024-04-06T02:42:24.544 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.742+0000 7faf2cff9700 1 -- 172.21.15.105:0/1367459555 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7faf24019c50 con 0x7faf30105470 2024-04-06T02:42:24.544 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.743+0000 7faf35bbf700 1 -- 172.21.15.105:0/1367459555 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf30191d00 con 0x7faf30105470 2024-04-06T02:42:24.544 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.743+0000 7faf2cff9700 1 --2- 172.21.15.105:0/1367459555 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7faf1803a4f0 0x7faf1803c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:24.544 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.743+0000 7faf2cff9700 1 -- 172.21.15.105:0/1367459555 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7faf2404c830 con 0x7faf30105470 2024-04-06T02:42:24.544 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.743+0000 7faf2effd700 1 --2- 172.21.15.105:0/1367459555 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7faf1803a4f0 0x7faf1803c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:24.544 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.743+0000 7faf2effd700 1 --2- 172.21.15.105:0/1367459555 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7faf1803a4f0 0x7faf1803c9b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7faf20006fd0 tx=0x7faf20006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:24.544 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.748+0000 7faf2cff9700 1 -- 172.21.15.105:0/1367459555 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7faf2402b700 con 0x7faf30105470 2024-04-06T02:42:24.545 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.955+0000 7faf35bbf700 1 -- 172.21.15.105:0/1367459555 --> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7faf300611d0 con 0x7faf1803a4f0 2024-04-06T02:42:24.545 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.962+0000 7faf2cff9700 1 -- 172.21.15.105:0/1367459555 <== mgr.14118 v2:172.21.15.105:6800/1394104199 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7faf300611d0 con 0x7faf1803a4f0 2024-04-06T02:42:24.545 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.964+0000 7faf35bbf700 1 -- 172.21.15.105:0/1367459555 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7faf1803a4f0 msgr2=0x7faf1803c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:24.545 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.964+0000 7faf35bbf700 1 --2- 172.21.15.105:0/1367459555 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7faf1803a4f0 0x7faf1803c9b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7faf20006fd0 tx=0x7faf20006e40 comp rx=0 tx=0).stop 2024-04-06T02:42:24.545 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.964+0000 7faf35bbf700 1 -- 172.21.15.105:0/1367459555 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf30105470 msgr2=0x7faf30197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:24.545 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.964+0000 7faf35bbf700 1 --2- 172.21.15.105:0/1367459555 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf30105470 0x7faf30197fe0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7faf24006ba0 tx=0x7faf24011b50 comp rx=0 tx=0).stop 2024-04-06T02:42:24.545 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.964+0000 7faf35bbf700 1 -- 172.21.15.105:0/1367459555 shutdown_connections 2024-04-06T02:42:24.545 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.964+0000 7faf35bbf700 1 --2- 172.21.15.105:0/1367459555 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7faf1803a4f0 0x7faf1803c9b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:24.545 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.964+0000 7faf35bbf700 1 --2- 172.21.15.105:0/1367459555 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf30105470 0x7faf30197fe0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:24.545 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.964+0000 7faf35bbf700 1 -- 172.21.15.105:0/1367459555 >> 172.21.15.105:0/1367459555 conn(0x7faf301009f0 msgr2=0x7faf3018ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:24.545 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.964+0000 7faf35bbf700 1 -- 172.21.15.105:0/1367459555 shutdown_connections 2024-04-06T02:42:24.545 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:23.964+0000 7faf35bbf700 1 -- 172.21.15.105:0/1367459555 wait complete. 2024-04-06T02:42:25.157 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:24 smithi105 ceph-mon[29228]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:42:25.157 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:24 smithi105 ceph-mon[29228]: Saving service alertmanager spec with placement count:1 2024-04-06T02:42:25.157 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:24 smithi105 ceph-mon[29228]: from='mgr.14118 172.21.15.105:0/768307854' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:25.157 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:24 smithi105 ceph-mon[29228]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:42:25.659 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.977+0000 7f77419f2700 1 Processor -- start 2024-04-06T02:42:25.664 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.977+0000 7f77419f2700 1 -- start start 2024-04-06T02:42:25.664 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.977+0000 7f77419f2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f773c106fe0 0x7f773c1093d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:25.664 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.977+0000 7f77419f2700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c074680 con 0x7f773c106fe0 2024-04-06T02:42:25.664 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.978+0000 7f773affd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f773c106fe0 0x7f773c1093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:25.664 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.978+0000 7f773affd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f773c106fe0 0x7f773c1093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53206/0 (socket says 172.21.15.105:53206) 2024-04-06T02:42:25.664 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.978+0000 7f773affd700 1 -- 172.21.15.105:0/3733473185 learned_addr learned my addr 172.21.15.105:0/3733473185 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:25.664 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.978+0000 7f773affd700 1 -- 172.21.15.105:0/3733473185 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f773c109910 con 0x7f773c106fe0 2024-04-06T02:42:25.664 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.978+0000 7f773affd700 1 --2- 172.21.15.105:0/3733473185 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f773c106fe0 0x7f773c1093d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f772c009e80 tx=0x7f772c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4efbff9dc8de796 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:25.664 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.978+0000 7f7739ffb700 1 -- 172.21.15.105:0/3733473185 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f772c00e070 con 0x7f773c106fe0 2024-04-06T02:42:25.664 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.978+0000 7f7739ffb700 1 -- 172.21.15.105:0/3733473185 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f772c00b8c0 con 0x7f773c106fe0 2024-04-06T02:42:25.665 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.978+0000 7f7739ffb700 1 -- 172.21.15.105:0/3733473185 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f772c0044d0 con 0x7f773c106fe0 2024-04-06T02:42:25.665 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.979+0000 7f77419f2700 1 -- 172.21.15.105:0/3733473185 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f773c106fe0 msgr2=0x7f773c1093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:25.665 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.979+0000 7f77419f2700 1 --2- 172.21.15.105:0/3733473185 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f773c106fe0 0x7f773c1093d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f772c009e80 tx=0x7f772c00b330 comp rx=0 tx=0).stop 2024-04-06T02:42:25.665 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.979+0000 7f77419f2700 1 -- 172.21.15.105:0/3733473185 shutdown_connections 2024-04-06T02:42:25.665 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.979+0000 7f77419f2700 1 --2- 172.21.15.105:0/3733473185 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f773c106fe0 0x7f773c1093d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:25.665 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.979+0000 7f77419f2700 1 -- 172.21.15.105:0/3733473185 >> 172.21.15.105:0/3733473185 conn(0x7f773c1009f0 msgr2=0x7f773c102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:25.665 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.979+0000 7f77419f2700 1 -- 172.21.15.105:0/3733473185 shutdown_connections 2024-04-06T02:42:25.665 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.979+0000 7f77419f2700 1 -- 172.21.15.105:0/3733473185 wait complete. 2024-04-06T02:42:25.665 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.979+0000 7f77419f2700 1 Processor -- start 2024-04-06T02:42:25.665 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.980+0000 7f77419f2700 1 -- start start 2024-04-06T02:42:25.665 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.980+0000 7f77419f2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f773c106fe0 0x7f773c19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:25.665 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.980+0000 7f77419f2700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c19a6e0 con 0x7f773c106fe0 2024-04-06T02:42:25.665 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.980+0000 7f773affd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f773c106fe0 0x7f773c19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:25.665 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.980+0000 7f773affd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f773c106fe0 0x7f773c19a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53218/0 (socket says 172.21.15.105:53218) 2024-04-06T02:42:25.665 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.980+0000 7f773affd700 1 -- 172.21.15.105:0/3417990662 learned_addr learned my addr 172.21.15.105:0/3417990662 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:25.665 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.980+0000 7f773affd700 1 -- 172.21.15.105:0/3417990662 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f772c009b30 con 0x7f773c106fe0 2024-04-06T02:42:25.666 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.980+0000 7f773affd700 1 --2- 172.21.15.105:0/3417990662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f773c106fe0 0x7f773c19a1a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f772c004070 tx=0x7f772c0049f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:25.666 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.980+0000 7f77409f0700 1 -- 172.21.15.105:0/3417990662 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f772c00e040 con 0x7f773c106fe0 2024-04-06T02:42:25.666 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.980+0000 7f77409f0700 1 -- 172.21.15.105:0/3417990662 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f772c00be00 con 0x7f773c106fe0 2024-04-06T02:42:25.666 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.980+0000 7f77419f2700 1 -- 172.21.15.105:0/3417990662 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f773c19a8e0 con 0x7f773c106fe0 2024-04-06T02:42:25.666 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.980+0000 7f77409f0700 1 -- 172.21.15.105:0/3417990662 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f772c012770 con 0x7f773c106fe0 2024-04-06T02:42:25.666 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.980+0000 7f77419f2700 1 -- 172.21.15.105:0/3417990662 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f773c19ade0 con 0x7f773c106fe0 2024-04-06T02:42:25.666 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.981+0000 7f77409f0700 1 -- 172.21.15.105:0/3417990662 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f772c019040 con 0x7f773c106fe0 2024-04-06T02:42:25.666 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.981+0000 7f77409f0700 1 --2- 172.21.15.105:0/3417990662 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f7728038320 0x7f772803a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:25.666 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.981+0000 7f77409f0700 1 -- 172.21.15.105:0/3417990662 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f772c04bf80 con 0x7f773c106fe0 2024-04-06T02:42:25.666 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.981+0000 7f77419f2700 1 -- 172.21.15.105:0/3417990662 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f773c0623c0 con 0x7f773c106fe0 2024-04-06T02:42:25.666 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.982+0000 7f773a7fc700 1 --2- 172.21.15.105:0/3417990662 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f7728038320 0x7f772803a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:25.666 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.982+0000 7f773a7fc700 1 --2- 172.21.15.105:0/3417990662 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f7728038320 0x7f772803a7e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7724006fd0 tx=0x7f7724006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:25.666 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:24.986+0000 7f77409f0700 1 -- 172.21.15.105:0/3417990662 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f772c01f080 con 0x7f773c106fe0 2024-04-06T02:42:25.666 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:25.172+0000 7f77419f2700 1 -- 172.21.15.105:0/3417990662 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f773c19d800 con 0x7f773c106fe0 2024-04-06T02:42:25.667 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:25.177+0000 7f77409f0700 1 -- 172.21.15.105:0/3417990662 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f772c011750 con 0x7f773c106fe0 2024-04-06T02:42:25.667 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:25.180+0000 7f77419f2700 1 -- 172.21.15.105:0/3417990662 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f7728038320 msgr2=0x7f772803a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:25.667 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:25.180+0000 7f77419f2700 1 --2- 172.21.15.105:0/3417990662 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f7728038320 0x7f772803a7e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7724006fd0 tx=0x7f7724006e40 comp rx=0 tx=0).stop 2024-04-06T02:42:25.667 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:25.180+0000 7f77419f2700 1 -- 172.21.15.105:0/3417990662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f773c106fe0 msgr2=0x7f773c19a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:25.667 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:25.180+0000 7f77419f2700 1 --2- 172.21.15.105:0/3417990662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f773c106fe0 0x7f773c19a1a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f772c004070 tx=0x7f772c0049f0 comp rx=0 tx=0).stop 2024-04-06T02:42:25.667 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:25.180+0000 7f77419f2700 1 -- 172.21.15.105:0/3417990662 shutdown_connections 2024-04-06T02:42:25.667 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:25.180+0000 7f77419f2700 1 --2- 172.21.15.105:0/3417990662 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f7728038320 0x7f772803a7e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:25.667 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:25.180+0000 7f77419f2700 1 --2- 172.21.15.105:0/3417990662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f773c106fe0 0x7f773c19a1a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:25.667 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:25.180+0000 7f77419f2700 1 -- 172.21.15.105:0/3417990662 >> 172.21.15.105:0/3417990662 conn(0x7f773c1009f0 msgr2=0x7f773c101720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:25.667 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:25.180+0000 7f77419f2700 1 -- 172.21.15.105:0/3417990662 shutdown_connections 2024-04-06T02:42:25.667 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:25.180+0000 7f77419f2700 1 -- 172.21.15.105:0/3417990662 wait complete. 2024-04-06T02:42:26.293 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:26 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/3417990662' entity='client.admin' 2024-04-06T02:42:26.854 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.094+0000 7fb92618d700 1 Processor -- start 2024-04-06T02:42:26.855 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.094+0000 7fb92618d700 1 -- start start 2024-04-06T02:42:26.855 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.094+0000 7fb92618d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb920106580 0x7fb9201069a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:26.855 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.094+0000 7fb92618d700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb920106f70 con 0x7fb920106580 2024-04-06T02:42:26.855 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.095+0000 7fb91f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb920106580 0x7fb9201069a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:26.855 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.095+0000 7fb91f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb920106580 0x7fb9201069a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53222/0 (socket says 172.21.15.105:53222) 2024-04-06T02:42:26.855 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.095+0000 7fb91f7fe700 1 -- 172.21.15.105:0/2333425939 learned_addr learned my addr 172.21.15.105:0/2333425939 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:26.855 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.095+0000 7fb91f7fe700 1 -- 172.21.15.105:0/2333425939 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb920107780 con 0x7fb920106580 2024-04-06T02:42:26.855 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.095+0000 7fb91f7fe700 1 --2- 172.21.15.105:0/2333425939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb920106580 0x7fb9201069a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb9100098a0 tx=0x7fb910009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6cffc9c4666ea0b server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:26.855 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.095+0000 7fb91effd700 1 -- 172.21.15.105:0/2333425939 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb910004180 con 0x7fb920106580 2024-04-06T02:42:26.855 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.095+0000 7fb91effd700 1 -- 172.21.15.105:0/2333425939 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb9100042e0 con 0x7fb920106580 2024-04-06T02:42:26.855 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.095+0000 7fb91effd700 1 -- 172.21.15.105:0/2333425939 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb910011af0 con 0x7fb920106580 2024-04-06T02:42:26.855 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.096+0000 7fb92618d700 1 -- 172.21.15.105:0/2333425939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb920106580 msgr2=0x7fb9201069a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:26.855 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.096+0000 7fb92618d700 1 --2- 172.21.15.105:0/2333425939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb920106580 0x7fb9201069a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb9100098a0 tx=0x7fb910009bb0 comp rx=0 tx=0).stop 2024-04-06T02:42:26.856 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.096+0000 7fb92618d700 1 -- 172.21.15.105:0/2333425939 shutdown_connections 2024-04-06T02:42:26.856 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.096+0000 7fb92618d700 1 --2- 172.21.15.105:0/2333425939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb920106580 0x7fb9201069a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:26.856 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.096+0000 7fb92618d700 1 -- 172.21.15.105:0/2333425939 >> 172.21.15.105:0/2333425939 conn(0x7fb920101b00 msgr2=0x7fb920103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:26.856 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.096+0000 7fb92618d700 1 -- 172.21.15.105:0/2333425939 shutdown_connections 2024-04-06T02:42:26.856 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.096+0000 7fb92618d700 1 -- 172.21.15.105:0/2333425939 wait complete. 2024-04-06T02:42:26.856 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.096+0000 7fb92618d700 1 Processor -- start 2024-04-06T02:42:26.856 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.096+0000 7fb92618d700 1 -- start start 2024-04-06T02:42:26.856 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.097+0000 7fb92618d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb920106580 0x7fb92007bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:26.856 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.097+0000 7fb92618d700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92007c170 con 0x7fb920106580 2024-04-06T02:42:26.856 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.097+0000 7fb91f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb920106580 0x7fb92007bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:26.856 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.097+0000 7fb91f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb920106580 0x7fb92007bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53232/0 (socket says 172.21.15.105:53232) 2024-04-06T02:42:26.856 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.097+0000 7fb91f7fe700 1 -- 172.21.15.105:0/1933357482 learned_addr learned my addr 172.21.15.105:0/1933357482 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:26.856 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.097+0000 7fb91f7fe700 1 -- 172.21.15.105:0/1933357482 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb910009550 con 0x7fb920106580 2024-04-06T02:42:26.856 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.097+0000 7fb91f7fe700 1 --2- 172.21.15.105:0/1933357482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb920106580 0x7fb92007bc30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb910011670 tx=0x7fb910011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:26.857 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.097+0000 7fb91d7fa700 1 -- 172.21.15.105:0/1933357482 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9100039f0 con 0x7fb920106580 2024-04-06T02:42:26.857 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.097+0000 7fb92618d700 1 -- 172.21.15.105:0/1933357482 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb92007a280 con 0x7fb920106580 2024-04-06T02:42:26.857 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.097+0000 7fb91d7fa700 1 -- 172.21.15.105:0/1933357482 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb910003b50 con 0x7fb920106580 2024-04-06T02:42:26.857 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.097+0000 7fb91d7fa700 1 -- 172.21.15.105:0/1933357482 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9100247c0 con 0x7fb920106580 2024-04-06T02:42:26.857 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.097+0000 7fb92618d700 1 -- 172.21.15.105:0/1933357482 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb92007a720 con 0x7fb920106580 2024-04-06T02:42:26.857 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.098+0000 7fb91d7fa700 1 -- 172.21.15.105:0/1933357482 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fb91002d3f0 con 0x7fb920106580 2024-04-06T02:42:26.857 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.098+0000 7fb92618d700 1 -- 172.21.15.105:0/1933357482 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9200623c0 con 0x7fb920106580 2024-04-06T02:42:26.857 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.098+0000 7fb91d7fa700 1 --2- 172.21.15.105:0/1933357482 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fb90c038320 0x7fb90c03a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:26.857 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.098+0000 7fb91d7fa700 1 -- 172.21.15.105:0/1933357482 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb91004c400 con 0x7fb920106580 2024-04-06T02:42:26.857 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.098+0000 7fb916ffd700 1 --2- 172.21.15.105:0/1933357482 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fb90c038320 0x7fb90c03a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:26.857 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.099+0000 7fb916ffd700 1 --2- 172.21.15.105:0/1933357482 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fb90c038320 0x7fb90c03a7e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb908006fd0 tx=0x7fb908006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:26.857 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.103+0000 7fb91d7fa700 1 -- 172.21.15.105:0/1933357482 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb910010970 con 0x7fb920106580 2024-04-06T02:42:26.857 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.291+0000 7fb92618d700 1 -- 172.21.15.105:0/1933357482 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fb92007af70 con 0x7fb920106580 2024-04-06T02:42:26.857 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.294+0000 7fb91d7fa700 1 -- 172.21.15.105:0/1933357482 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fb910020220 con 0x7fb920106580 2024-04-06T02:42:26.857 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.297+0000 7fb92618d700 1 -- 172.21.15.105:0/1933357482 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fb90c038320 msgr2=0x7fb90c03a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:26.858 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.297+0000 7fb92618d700 1 --2- 172.21.15.105:0/1933357482 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fb90c038320 0x7fb90c03a7e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb908006fd0 tx=0x7fb908006e40 comp rx=0 tx=0).stop 2024-04-06T02:42:26.858 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.297+0000 7fb92618d700 1 -- 172.21.15.105:0/1933357482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb920106580 msgr2=0x7fb92007bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:26.858 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.297+0000 7fb92618d700 1 --2- 172.21.15.105:0/1933357482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb920106580 0x7fb92007bc30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb910011670 tx=0x7fb910011b50 comp rx=0 tx=0).stop 2024-04-06T02:42:26.858 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.297+0000 7fb92618d700 1 -- 172.21.15.105:0/1933357482 shutdown_connections 2024-04-06T02:42:26.858 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.297+0000 7fb92618d700 1 --2- 172.21.15.105:0/1933357482 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fb90c038320 0x7fb90c03a7e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:26.858 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.297+0000 7fb92618d700 1 --2- 172.21.15.105:0/1933357482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb920106580 0x7fb92007bc30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:26.858 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.297+0000 7fb92618d700 1 -- 172.21.15.105:0/1933357482 >> 172.21.15.105:0/1933357482 conn(0x7fb920101b00 msgr2=0x7fb9201031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:26.858 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.297+0000 7fb92618d700 1 -- 172.21.15.105:0/1933357482 shutdown_connections 2024-04-06T02:42:26.858 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:26.297+0000 7fb92618d700 1 -- 172.21.15.105:0/1933357482 wait complete. 2024-04-06T02:42:26.858 INFO:teuthology.orchestra.run.smithi105.stdout:Enabling the dashboard module... 2024-04-06T02:42:27.547 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:27 smithi105 ceph-mon[29228]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:42:27.547 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:27 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/1933357482' entity='client.admin' 2024-04-06T02:42:28.610 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:28 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/4265204795' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-06T02:42:28.944 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.313+0000 7f88ac0b1700 1 Processor -- start 2024-04-06T02:42:28.944 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.313+0000 7f88ac0b1700 1 -- start start 2024-04-06T02:42:28.944 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.314+0000 7f88ac0b1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88a407be80 0x7f88a407a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:28.944 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.314+0000 7f88ac0b1700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a407a8b0 con 0x7f88a407be80 2024-04-06T02:42:28.944 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.314+0000 7f88a9e4d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88a407be80 0x7f88a407a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:28.944 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.314+0000 7f88a9e4d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88a407be80 0x7f88a407a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53240/0 (socket says 172.21.15.105:53240) 2024-04-06T02:42:28.944 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.314+0000 7f88a9e4d700 1 -- 172.21.15.105:0/3116716484 learned_addr learned my addr 172.21.15.105:0/3116716484 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:28.944 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.314+0000 7f88a9e4d700 1 -- 172.21.15.105:0/3116716484 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a407a9f0 con 0x7f88a407be80 2024-04-06T02:42:28.944 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.314+0000 7f88a9e4d700 1 --2- 172.21.15.105:0/3116716484 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88a407be80 0x7f88a407a2e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f88a00098a0 tx=0x7f88a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9292cac87c359959 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:28.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.314+0000 7f88a8e4b700 1 -- 172.21.15.105:0/3116716484 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88a0004180 con 0x7f88a407be80 2024-04-06T02:42:28.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.314+0000 7f88a8e4b700 1 -- 172.21.15.105:0/3116716484 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f88a00042e0 con 0x7f88a407be80 2024-04-06T02:42:28.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.314+0000 7f88a8e4b700 1 -- 172.21.15.105:0/3116716484 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88a0011af0 con 0x7f88a407be80 2024-04-06T02:42:28.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.315+0000 7f88ac0b1700 1 -- 172.21.15.105:0/3116716484 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88a407be80 msgr2=0x7f88a407a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:28.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.315+0000 7f88ac0b1700 1 --2- 172.21.15.105:0/3116716484 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88a407be80 0x7f88a407a2e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f88a00098a0 tx=0x7f88a0009bb0 comp rx=0 tx=0).stop 2024-04-06T02:42:28.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.315+0000 7f88ac0b1700 1 -- 172.21.15.105:0/3116716484 shutdown_connections 2024-04-06T02:42:28.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.315+0000 7f88ac0b1700 1 --2- 172.21.15.105:0/3116716484 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88a407be80 0x7f88a407a2e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:28.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.315+0000 7f88ac0b1700 1 -- 172.21.15.105:0/3116716484 >> 172.21.15.105:0/3116716484 conn(0x7f88a4101b40 msgr2=0x7f88a4103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:28.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.315+0000 7f88ac0b1700 1 -- 172.21.15.105:0/3116716484 shutdown_connections 2024-04-06T02:42:28.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.315+0000 7f88ac0b1700 1 -- 172.21.15.105:0/3116716484 wait complete. 2024-04-06T02:42:28.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.316+0000 7f88ac0b1700 1 Processor -- start 2024-04-06T02:42:28.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.316+0000 7f88ac0b1700 1 -- start start 2024-04-06T02:42:28.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.316+0000 7f88ac0b1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88a407be80 0x7f88a419c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:28.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.316+0000 7f88ac0b1700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a419c9a0 con 0x7f88a407be80 2024-04-06T02:42:28.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.316+0000 7f88a9e4d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88a407be80 0x7f88a419c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:28.946 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.316+0000 7f88a9e4d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88a407be80 0x7f88a419c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53244/0 (socket says 172.21.15.105:53244) 2024-04-06T02:42:28.946 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.316+0000 7f88a9e4d700 1 -- 172.21.15.105:0/4265204795 learned_addr learned my addr 172.21.15.105:0/4265204795 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:28.946 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.316+0000 7f88a9e4d700 1 -- 172.21.15.105:0/4265204795 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a0009550 con 0x7f88a407be80 2024-04-06T02:42:28.946 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.316+0000 7f88a9e4d700 1 --2- 172.21.15.105:0/4265204795 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88a407be80 0x7f88a419c460 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f88a0011670 tx=0x7f88a0011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:28.946 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.317+0000 7f889affd700 1 -- 172.21.15.105:0/4265204795 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88a00039f0 con 0x7f88a407be80 2024-04-06T02:42:28.946 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.317+0000 7f889affd700 1 -- 172.21.15.105:0/4265204795 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f88a0003b50 con 0x7f88a407be80 2024-04-06T02:42:28.946 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.317+0000 7f88ac0b1700 1 -- 172.21.15.105:0/4265204795 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a419cba0 con 0x7f88a407be80 2024-04-06T02:42:28.946 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.317+0000 7f889affd700 1 -- 172.21.15.105:0/4265204795 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88a00247c0 con 0x7f88a407be80 2024-04-06T02:42:28.946 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.317+0000 7f88ac0b1700 1 -- 172.21.15.105:0/4265204795 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a419d040 con 0x7f88a407be80 2024-04-06T02:42:28.946 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.317+0000 7f889affd700 1 -- 172.21.15.105:0/4265204795 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f88a002d3f0 con 0x7f88a407be80 2024-04-06T02:42:28.946 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.318+0000 7f88ac0b1700 1 -- 172.21.15.105:0/4265204795 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a410c920 con 0x7f88a407be80 2024-04-06T02:42:28.946 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.318+0000 7f889affd700 1 --2- 172.21.15.105:0/4265204795 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f8890038370 0x7f889003a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:28.946 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.318+0000 7f889affd700 1 -- 172.21.15.105:0/4265204795 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f88a004c390 con 0x7f88a407be80 2024-04-06T02:42:28.946 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.318+0000 7f88a964c700 1 --2- 172.21.15.105:0/4265204795 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f8890038370 0x7f889003a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:28.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.318+0000 7f88a964c700 1 --2- 172.21.15.105:0/4265204795 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f8890038370 0x7f889003a830 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8894006fd0 tx=0x7f8894006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:28.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.323+0000 7f889affd700 1 -- 172.21.15.105:0/4265204795 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f88a0010970 con 0x7f88a407be80 2024-04-06T02:42:28.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:27.562+0000 7f88ac0b1700 1 -- 172.21.15.105:0/4265204795 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f88a419fe80 con 0x7f88a407be80 2024-04-06T02:42:28.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:28.299+0000 7f889affd700 1 -- 172.21.15.105:0/4265204795 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 9) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f88a001ac00 con 0x7f88a407be80 2024-04-06T02:42:28.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:28.301+0000 7f889affd700 1 -- 172.21.15.105:0/4265204795 <== mon.0 v2:172.21.15.105:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f88a001a950 con 0x7f88a407be80 2024-04-06T02:42:28.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:28.303+0000 7f88ac0b1700 1 -- 172.21.15.105:0/4265204795 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f8890038370 msgr2=0x7f889003a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:28.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:28.303+0000 7f88ac0b1700 1 --2- 172.21.15.105:0/4265204795 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f8890038370 0x7f889003a830 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8894006fd0 tx=0x7f8894006e40 comp rx=0 tx=0).stop 2024-04-06T02:42:28.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:28.303+0000 7f88ac0b1700 1 -- 172.21.15.105:0/4265204795 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88a407be80 msgr2=0x7f88a419c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:28.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:28.303+0000 7f88ac0b1700 1 --2- 172.21.15.105:0/4265204795 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88a407be80 0x7f88a419c460 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f88a0011670 tx=0x7f88a0011b50 comp rx=0 tx=0).stop 2024-04-06T02:42:28.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:28.303+0000 7f88ac0b1700 1 -- 172.21.15.105:0/4265204795 shutdown_connections 2024-04-06T02:42:28.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:28.303+0000 7f88ac0b1700 1 --2- 172.21.15.105:0/4265204795 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f8890038370 0x7f889003a830 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:28.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:28.303+0000 7f88ac0b1700 1 --2- 172.21.15.105:0/4265204795 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88a407be80 0x7f88a419c460 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:28.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:28.303+0000 7f88ac0b1700 1 -- 172.21.15.105:0/4265204795 >> 172.21.15.105:0/4265204795 conn(0x7f88a4101b40 msgr2=0x7f88a41031a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:28.948 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:28.304+0000 7f88ac0b1700 1 -- 172.21.15.105:0/4265204795 shutdown_connections 2024-04-06T02:42:28.948 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:28.304+0000 7f88ac0b1700 1 -- 172.21.15.105:0/4265204795 wait complete. 2024-04-06T02:42:29.599 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:29 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/4265204795' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-06T02:42:29.599 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:29 smithi105 ceph-mon[29228]: mgrmap e9: smithi105.vqqcpx(active, since 26s) 2024-04-06T02:42:30.242 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout { 2024-04-06T02:42:30.243 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-06T02:42:30.243 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-06T02:42:30.243 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "active_name": "smithi105.vqqcpx", 2024-04-06T02:42:30.243 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-06T02:42:30.243 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout } 2024-04-06T02:42:30.243 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.414+0000 7fb54ac99700 1 Processor -- start 2024-04-06T02:42:30.243 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.414+0000 7fb54ac99700 1 -- start start 2024-04-06T02:42:30.243 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.414+0000 7fb54ac99700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb544106560 0x7fb544106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:30.243 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.415+0000 7fb54ac99700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb544106f50 con 0x7fb544106560 2024-04-06T02:42:30.243 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.415+0000 7fb548a35700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb544106560 0x7fb544106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:30.243 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.415+0000 7fb548a35700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb544106560 0x7fb544106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53288/0 (socket says 172.21.15.105:53288) 2024-04-06T02:42:30.243 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.415+0000 7fb548a35700 1 -- 172.21.15.105:0/3578157670 learned_addr learned my addr 172.21.15.105:0/3578157670 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:30.243 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.415+0000 7fb548a35700 1 -- 172.21.15.105:0/3578157670 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb544107760 con 0x7fb544106560 2024-04-06T02:42:30.243 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.415+0000 7fb548a35700 1 --2- 172.21.15.105:0/3578157670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb544106560 0x7fb544106980 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb5380098a0 tx=0x7fb538009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84f1dcdcb1f10126 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:30.244 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.415+0000 7fb5437fe700 1 -- 172.21.15.105:0/3578157670 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb538004180 con 0x7fb544106560 2024-04-06T02:42:30.244 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.415+0000 7fb5437fe700 1 -- 172.21.15.105:0/3578157670 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb5380042e0 con 0x7fb544106560 2024-04-06T02:42:30.244 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.415+0000 7fb5437fe700 1 -- 172.21.15.105:0/3578157670 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb538011af0 con 0x7fb544106560 2024-04-06T02:42:30.244 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.416+0000 7fb54ac99700 1 -- 172.21.15.105:0/3578157670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb544106560 msgr2=0x7fb544106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:30.244 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.416+0000 7fb54ac99700 1 --2- 172.21.15.105:0/3578157670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb544106560 0x7fb544106980 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb5380098a0 tx=0x7fb538009bb0 comp rx=0 tx=0).stop 2024-04-06T02:42:30.244 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.416+0000 7fb54ac99700 1 -- 172.21.15.105:0/3578157670 shutdown_connections 2024-04-06T02:42:30.244 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.416+0000 7fb54ac99700 1 --2- 172.21.15.105:0/3578157670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb544106560 0x7fb544106980 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:30.244 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.416+0000 7fb54ac99700 1 -- 172.21.15.105:0/3578157670 >> 172.21.15.105:0/3578157670 conn(0x7fb544101b40 msgr2=0x7fb544103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:30.244 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.416+0000 7fb54ac99700 1 -- 172.21.15.105:0/3578157670 shutdown_connections 2024-04-06T02:42:30.244 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.416+0000 7fb54ac99700 1 -- 172.21.15.105:0/3578157670 wait complete. 2024-04-06T02:42:30.244 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.417+0000 7fb54ac99700 1 Processor -- start 2024-04-06T02:42:30.244 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.417+0000 7fb54ac99700 1 -- start start 2024-04-06T02:42:30.244 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.417+0000 7fb54ac99700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb544106560 0x7fb54419a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:30.244 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.417+0000 7fb54ac99700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54419a730 con 0x7fb544106560 2024-04-06T02:42:30.244 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.417+0000 7fb548a35700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb544106560 0x7fb54419a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:30.245 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.417+0000 7fb548a35700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb544106560 0x7fb54419a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53292/0 (socket says 172.21.15.105:53292) 2024-04-06T02:42:30.245 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.417+0000 7fb548a35700 1 -- 172.21.15.105:0/3947425874 learned_addr learned my addr 172.21.15.105:0/3947425874 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:30.245 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.417+0000 7fb548a35700 1 -- 172.21.15.105:0/3947425874 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb538009550 con 0x7fb544106560 2024-04-06T02:42:30.245 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.418+0000 7fb548a35700 1 --2- 172.21.15.105:0/3947425874 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb544106560 0x7fb54419a1f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb538006ba0 tx=0x7fb538011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:30.245 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.418+0000 7fb541ffb700 1 -- 172.21.15.105:0/3947425874 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb538003b00 con 0x7fb544106560 2024-04-06T02:42:30.245 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.418+0000 7fb541ffb700 1 -- 172.21.15.105:0/3947425874 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb53801a8e0 con 0x7fb544106560 2024-04-06T02:42:30.245 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.418+0000 7fb541ffb700 1 -- 172.21.15.105:0/3947425874 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb538019990 con 0x7fb544106560 2024-04-06T02:42:30.245 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.418+0000 7fb54ac99700 1 -- 172.21.15.105:0/3947425874 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb54419a930 con 0x7fb544106560 2024-04-06T02:42:30.245 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.418+0000 7fb54ac99700 1 -- 172.21.15.105:0/3947425874 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb54419add0 con 0x7fb544106560 2024-04-06T02:42:30.245 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.419+0000 7fb541ffb700 1 -- 172.21.15.105:0/3947425874 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 9) v1 ==== 45296+0+0 (secure 0 0 0) 0x7fb538019c50 con 0x7fb544106560 2024-04-06T02:42:30.245 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.419+0000 7fb54ac99700 1 -- 172.21.15.105:0/3947425874 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5440623c0 con 0x7fb544106560 2024-04-06T02:42:30.245 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.419+0000 7fb541ffb700 1 --2- 172.21.15.105:0/3947425874 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fb534038320 0x7fb53403a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:30.245 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.419+0000 7fb541ffb700 1 -- 172.21.15.105:0/3947425874 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb53804c5d0 con 0x7fb544106560 2024-04-06T02:42:30.245 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.419+0000 7fb543fff700 1 -- 172.21.15.105:0/3947425874 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fb534038320 msgr2=0x7fb53403a7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/1394104199 2024-04-06T02:42:30.245 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.419+0000 7fb543fff700 1 --2- 172.21.15.105:0/3947425874 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fb534038320 0x7fb53403a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T02:42:30.246 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.429+0000 7fb541ffb700 1 -- 172.21.15.105:0/3947425874 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb538018530 con 0x7fb544106560 2024-04-06T02:42:30.246 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.620+0000 7fb543fff700 1 -- 172.21.15.105:0/3947425874 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fb534038320 msgr2=0x7fb53403a7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/1394104199 2024-04-06T02:42:30.246 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.620+0000 7fb543fff700 1 --2- 172.21.15.105:0/3947425874 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fb534038320 0x7fb53403a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T02:42:30.246 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.680+0000 7fb54ac99700 1 -- 172.21.15.105:0/3947425874 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fb54410a990 con 0x7fb544106560 2024-04-06T02:42:30.246 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.681+0000 7fb541ffb700 1 -- 172.21.15.105:0/3947425874 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7fb538018350 con 0x7fb544106560 2024-04-06T02:42:30.246 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.683+0000 7fb54ac99700 1 -- 172.21.15.105:0/3947425874 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fb534038320 msgr2=0x7fb53403a7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T02:42:30.246 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.683+0000 7fb54ac99700 1 --2- 172.21.15.105:0/3947425874 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fb534038320 0x7fb53403a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:30.246 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.683+0000 7fb54ac99700 1 -- 172.21.15.105:0/3947425874 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb544106560 msgr2=0x7fb54419a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:30.246 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.683+0000 7fb54ac99700 1 --2- 172.21.15.105:0/3947425874 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb544106560 0x7fb54419a1f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb538006ba0 tx=0x7fb538011b50 comp rx=0 tx=0).stop 2024-04-06T02:42:30.247 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.683+0000 7fb54ac99700 1 -- 172.21.15.105:0/3947425874 shutdown_connections 2024-04-06T02:42:30.247 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.683+0000 7fb54ac99700 1 --2- 172.21.15.105:0/3947425874 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7fb534038320 0x7fb53403a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:30.247 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.683+0000 7fb54ac99700 1 --2- 172.21.15.105:0/3947425874 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb544106560 0x7fb54419a1f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:30.247 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.683+0000 7fb54ac99700 1 -- 172.21.15.105:0/3947425874 >> 172.21.15.105:0/3947425874 conn(0x7fb544101b40 msgr2=0x7fb544102890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:30.247 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.683+0000 7fb54ac99700 1 -- 172.21.15.105:0/3947425874 shutdown_connections 2024-04-06T02:42:30.247 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:29.683+0000 7fb54ac99700 1 -- 172.21.15.105:0/3947425874 wait complete. 2024-04-06T02:42:30.247 INFO:teuthology.orchestra.run.smithi105.stdout:Waiting for the mgr to restart... 2024-04-06T02:42:30.247 INFO:teuthology.orchestra.run.smithi105.stdout:Waiting for mgr epoch 9... 2024-04-06T02:42:30.496 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:30 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/3947425874' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-06T02:42:35.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:35 smithi105 ceph-mon[29228]: Active manager daemon smithi105.vqqcpx restarted 2024-04-06T02:42:35.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:35 smithi105 ceph-mon[29228]: Activating manager daemon smithi105.vqqcpx 2024-04-06T02:42:35.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:35 smithi105 ceph-mon[29228]: osdmap e3: 0 total, 0 up, 0 in 2024-04-06T02:42:35.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:35 smithi105 ceph-mon[29228]: mgrmap e10: smithi105.vqqcpx(active, starting, since 0.0032339s) 2024-04-06T02:42:35.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:35 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-04-06T02:42:35.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:35 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mgr metadata", "who": "smithi105.vqqcpx", "id": "smithi105.vqqcpx"}]: dispatch 2024-04-06T02:42:35.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:35 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T02:42:35.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:35 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T02:42:35.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:35 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T02:42:35.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:35 smithi105 ceph-mon[29228]: Manager daemon smithi105.vqqcpx is now available 2024-04-06T02:42:35.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:35 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:42:35.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:35 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:42:36.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:36 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi105.vqqcpx/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:42:36.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:36 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi105.vqqcpx/trash_purge_schedule"}]: dispatch 2024-04-06T02:42:36.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:36 smithi105 ceph-mon[29228]: mgrmap e11: smithi105.vqqcpx(active, since 1.00561s) 2024-04-06T02:42:36.806 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout { 2024-04-06T02:42:36.806 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-06T02:42:36.806 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-06T02:42:36.806 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout } 2024-04-06T02:42:36.806 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.722+0000 7f18c53d0700 1 Processor -- start 2024-04-06T02:42:36.806 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.722+0000 7f18c53d0700 1 -- start start 2024-04-06T02:42:36.806 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.722+0000 7f18c53d0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18c0104e70 0x7f18c0105290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:36.806 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.722+0000 7f18c53d0700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c00745e0 con 0x7f18c0104e70 2024-04-06T02:42:36.806 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.722+0000 7f18beffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18c0104e70 0x7f18c0105290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:36.806 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.722+0000 7f18beffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18c0104e70 0x7f18c0105290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:45774/0 (socket says 172.21.15.105:45774) 2024-04-06T02:42:36.807 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.722+0000 7f18beffd700 1 -- 172.21.15.105:0/1103742717 learned_addr learned my addr 172.21.15.105:0/1103742717 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:36.807 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.722+0000 7f18beffd700 1 -- 172.21.15.105:0/1103742717 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c0074720 con 0x7f18c0104e70 2024-04-06T02:42:36.807 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.723+0000 7f18beffd700 1 --2- 172.21.15.105:0/1103742717 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18c0104e70 0x7f18c0105290 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f18b00098a0 tx=0x7f18b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ebabe547161a9b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:36.807 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.723+0000 7f18bdffb700 1 -- 172.21.15.105:0/1103742717 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18b0004180 con 0x7f18c0104e70 2024-04-06T02:42:36.807 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.723+0000 7f18bdffb700 1 -- 172.21.15.105:0/1103742717 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f18b00042e0 con 0x7f18c0104e70 2024-04-06T02:42:36.807 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.723+0000 7f18bdffb700 1 -- 172.21.15.105:0/1103742717 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18b0011af0 con 0x7f18c0104e70 2024-04-06T02:42:36.807 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.723+0000 7f18c53d0700 1 -- 172.21.15.105:0/1103742717 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18c0104e70 msgr2=0x7f18c0105290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:36.807 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.723+0000 7f18c53d0700 1 --2- 172.21.15.105:0/1103742717 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18c0104e70 0x7f18c0105290 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f18b00098a0 tx=0x7f18b0009bb0 comp rx=0 tx=0).stop 2024-04-06T02:42:36.807 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.724+0000 7f18c53d0700 1 -- 172.21.15.105:0/1103742717 shutdown_connections 2024-04-06T02:42:36.807 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.724+0000 7f18c53d0700 1 --2- 172.21.15.105:0/1103742717 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18c0104e70 0x7f18c0105290 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:36.807 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.724+0000 7f18c53d0700 1 -- 172.21.15.105:0/1103742717 >> 172.21.15.105:0/1103742717 conn(0x7f18c01009f0 msgr2=0x7f18c0102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:36.807 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.724+0000 7f18c53d0700 1 -- 172.21.15.105:0/1103742717 shutdown_connections 2024-04-06T02:42:36.807 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.724+0000 7f18c53d0700 1 -- 172.21.15.105:0/1103742717 wait complete. 2024-04-06T02:42:36.807 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.724+0000 7f18c53d0700 1 Processor -- start 2024-04-06T02:42:36.808 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.724+0000 7f18c53d0700 1 -- start start 2024-04-06T02:42:36.808 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.725+0000 7f18c53d0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18c0104e70 0x7f18c019c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:36.808 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.725+0000 7f18c53d0700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c019c8f0 con 0x7f18c0104e70 2024-04-06T02:42:36.808 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.725+0000 7f18beffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18c0104e70 0x7f18c019c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:36.808 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.725+0000 7f18beffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18c0104e70 0x7f18c019c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:45784/0 (socket says 172.21.15.105:45784) 2024-04-06T02:42:36.808 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.725+0000 7f18beffd700 1 -- 172.21.15.105:0/4220077132 learned_addr learned my addr 172.21.15.105:0/4220077132 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:36.808 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.725+0000 7f18beffd700 1 -- 172.21.15.105:0/4220077132 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b0009550 con 0x7f18c0104e70 2024-04-06T02:42:36.808 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.725+0000 7f18beffd700 1 --2- 172.21.15.105:0/4220077132 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18c0104e70 0x7f18c019c3b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f18b0006ba0 tx=0x7f18b0011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:36.808 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.725+0000 7f18a7fff700 1 -- 172.21.15.105:0/4220077132 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18b0003b00 con 0x7f18c0104e70 2024-04-06T02:42:36.808 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.725+0000 7f18c53d0700 1 -- 172.21.15.105:0/4220077132 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c019caf0 con 0x7f18c0104e70 2024-04-06T02:42:36.808 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.725+0000 7f18a7fff700 1 -- 172.21.15.105:0/4220077132 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f18b001a8e0 con 0x7f18c0104e70 2024-04-06T02:42:36.808 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.725+0000 7f18a7fff700 1 -- 172.21.15.105:0/4220077132 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18b00199c0 con 0x7f18c0104e70 2024-04-06T02:42:36.808 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.725+0000 7f18c53d0700 1 -- 172.21.15.105:0/4220077132 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c019cf90 con 0x7f18c0104e70 2024-04-06T02:42:36.809 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.726+0000 7f18a7fff700 1 -- 172.21.15.105:0/4220077132 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 9) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f18b0019c30 con 0x7f18c0104e70 2024-04-06T02:42:36.809 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.726+0000 7f18a7fff700 1 --2- 172.21.15.105:0/4220077132 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f18a8038370 0x7f18a803a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:36.809 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.726+0000 7f18a7fff700 1 -- 172.21.15.105:0/4220077132 --> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18a803af40 con 0x7f18a8038370 2024-04-06T02:42:36.809 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.727+0000 7f18a7fff700 1 -- 172.21.15.105:0/4220077132 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f18b004c5b0 con 0x7f18c0104e70 2024-04-06T02:42:36.809 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.727+0000 7f18be7fc700 1 -- 172.21.15.105:0/4220077132 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f18a8038370 msgr2=0x7f18a803a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/1394104199 2024-04-06T02:42:36.809 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.727+0000 7f18be7fc700 1 --2- 172.21.15.105:0/4220077132 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f18a8038370 0x7f18a803a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T02:42:36.809 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.928+0000 7f18be7fc700 1 -- 172.21.15.105:0/4220077132 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f18a8038370 msgr2=0x7f18a803a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/1394104199 2024-04-06T02:42:36.809 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:30.928+0000 7f18be7fc700 1 --2- 172.21.15.105:0/4220077132 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f18a8038370 0x7f18a803a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T02:42:36.809 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:31.329+0000 7f18be7fc700 1 -- 172.21.15.105:0/4220077132 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f18a8038370 msgr2=0x7f18a803a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/1394104199 2024-04-06T02:42:36.809 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:31.329+0000 7f18be7fc700 1 --2- 172.21.15.105:0/4220077132 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f18a8038370 0x7f18a803a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-06T02:42:36.809 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:32.129+0000 7f18be7fc700 1 -- 172.21.15.105:0/4220077132 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f18a8038370 msgr2=0x7f18a803a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/1394104199 2024-04-06T02:42:36.809 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:32.129+0000 7f18be7fc700 1 --2- 172.21.15.105:0/4220077132 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f18a8038370 0x7f18a803a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-06T02:42:36.810 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:33.730+0000 7f18be7fc700 1 -- 172.21.15.105:0/4220077132 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f18a8038370 msgr2=0x7f18a803a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/1394104199 2024-04-06T02:42:36.810 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:33.730+0000 7f18be7fc700 1 --2- 172.21.15.105:0/4220077132 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f18a8038370 0x7f18a803a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-06T02:42:36.810 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:35.033+0000 7f18a7fff700 1 -- 172.21.15.105:0/4220077132 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mgrmap(e 10) v1 ==== 45063+0+0 (secure 0 0 0) 0x7f18b002d620 con 0x7f18c0104e70 2024-04-06T02:42:36.810 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:35.033+0000 7f18a7fff700 1 -- 172.21.15.105:0/4220077132 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f18a8038370 msgr2=0x7f18a803a830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T02:42:36.810 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:35.033+0000 7f18a7fff700 1 --2- 172.21.15.105:0/4220077132 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f18a8038370 0x7f18a803a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:36.810 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:36.034+0000 7f18a7fff700 1 -- 172.21.15.105:0/4220077132 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 11) v1 ==== 45190+0+0 (secure 0 0 0) 0x7f18b000ed40 con 0x7f18c0104e70 2024-04-06T02:42:36.810 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:36.034+0000 7f18a7fff700 1 --2- 172.21.15.105:0/4220077132 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f18a803bf00 0x7f18a803e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:36.810 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:36.034+0000 7f18a7fff700 1 -- 172.21.15.105:0/4220077132 --> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18a803af40 con 0x7f18a803bf00 2024-04-06T02:42:36.810 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:36.035+0000 7f18be7fc700 1 --2- 172.21.15.105:0/4220077132 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f18a803bf00 0x7f18a803e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:36.810 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:36.035+0000 7f18be7fc700 1 --2- 172.21.15.105:0/4220077132 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f18a803bf00 0x7f18a803e2f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f18b4003a10 tx=0x7f18b4009270 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:36.810 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:36.036+0000 7f18a7fff700 1 -- 172.21.15.105:0/4220077132 <== mgr.14162 v2:172.21.15.105:6800/1949998385 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f18a803af40 con 0x7f18a803bf00 2024-04-06T02:42:36.810 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:36.042+0000 7f18c53d0700 1 -- 172.21.15.105:0/4220077132 --> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f18c019d6f0 con 0x7f18a803bf00 2024-04-06T02:42:36.810 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:36.042+0000 7f18a7fff700 1 -- 172.21.15.105:0/4220077132 <== mgr.14162 v2:172.21.15.105:6800/1949998385 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f18c019d6f0 con 0x7f18a803bf00 2024-04-06T02:42:36.811 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:36.042+0000 7f18c53d0700 1 -- 172.21.15.105:0/4220077132 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f18a803bf00 msgr2=0x7f18a803e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:36.811 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:36.042+0000 7f18c53d0700 1 --2- 172.21.15.105:0/4220077132 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f18a803bf00 0x7f18a803e2f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f18b4003a10 tx=0x7f18b4009270 comp rx=0 tx=0).stop 2024-04-06T02:42:36.811 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:36.042+0000 7f18c53d0700 1 -- 172.21.15.105:0/4220077132 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18c0104e70 msgr2=0x7f18c019c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:36.811 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:36.042+0000 7f18c53d0700 1 --2- 172.21.15.105:0/4220077132 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18c0104e70 0x7f18c019c3b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f18b0006ba0 tx=0x7f18b0011b50 comp rx=0 tx=0).stop 2024-04-06T02:42:36.811 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:36.042+0000 7f18c53d0700 1 -- 172.21.15.105:0/4220077132 shutdown_connections 2024-04-06T02:42:36.811 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:36.042+0000 7f18c53d0700 1 --2- 172.21.15.105:0/4220077132 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18c0104e70 0x7f18c019c3b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:36.811 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:36.042+0000 7f18c53d0700 1 --2- 172.21.15.105:0/4220077132 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f18a803bf00 0x7f18a803e2f0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:36.811 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:36.042+0000 7f18c53d0700 1 --2- 172.21.15.105:0/4220077132 >> [v2:172.21.15.105:6800/1394104199,v1:172.21.15.105:6801/1394104199] conn(0x7f18a8038370 0x7f18a803a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:36.811 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:36.042+0000 7f18c53d0700 1 -- 172.21.15.105:0/4220077132 >> 172.21.15.105:0/4220077132 conn(0x7f18c01009f0 msgr2=0x7f18c0190a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:36.811 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:36.043+0000 7f18c53d0700 1 -- 172.21.15.105:0/4220077132 shutdown_connections 2024-04-06T02:42:36.811 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:36.043+0000 7f18c53d0700 1 -- 172.21.15.105:0/4220077132 wait complete. 2024-04-06T02:42:36.811 INFO:teuthology.orchestra.run.smithi105.stdout:mgr epoch 9 is available 2024-04-06T02:42:36.811 INFO:teuthology.orchestra.run.smithi105.stdout:Generating a dashboard self-signed certificate... 2024-04-06T02:42:37.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:37 smithi105 ceph-mon[29228]: [06/Apr/2024:02:42:35] ENGINE Bus STARTING 2024-04-06T02:42:37.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:37 smithi105 ceph-mon[29228]: [06/Apr/2024:02:42:35] ENGINE Serving on http://172.21.15.105:8765 2024-04-06T02:42:37.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:37 smithi105 ceph-mon[29228]: [06/Apr/2024:02:42:35] ENGINE Serving on https://172.21.15.105:7150 2024-04-06T02:42:37.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:37 smithi105 ceph-mon[29228]: [06/Apr/2024:02:42:35] ENGINE Bus STARTED 2024-04-06T02:42:37.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:37 smithi105 ceph-mon[29228]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-06T02:42:37.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:37 smithi105 ceph-mon[29228]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-06T02:42:37.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:37 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:38.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-06T02:42:38.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.280+0000 7fa2ab934700 1 Processor -- start 2024-04-06T02:42:38.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.280+0000 7fa2ab934700 1 -- start start 2024-04-06T02:42:38.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.281+0000 7fa2ab934700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2a4105470 0x7fa2a4105890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:38.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.281+0000 7fa2ab934700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a4105e60 con 0x7fa2a4105470 2024-04-06T02:42:38.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.281+0000 7fa2a96d0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2a4105470 0x7fa2a4105890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:38.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.281+0000 7fa2a96d0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2a4105470 0x7fa2a4105890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:45870/0 (socket says 172.21.15.105:45870) 2024-04-06T02:42:38.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.281+0000 7fa2a96d0700 1 -- 172.21.15.105:0/1855245010 learned_addr learned my addr 172.21.15.105:0/1855245010 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:38.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.281+0000 7fa2a96d0700 1 -- 172.21.15.105:0/1855245010 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a4106670 con 0x7fa2a4105470 2024-04-06T02:42:38.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.281+0000 7fa2a96d0700 1 --2- 172.21.15.105:0/1855245010 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2a4105470 0x7fa2a4105890 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa29c009e80 tx=0x7fa29c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70123171b49327c server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:38.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.281+0000 7fa29bfff700 1 -- 172.21.15.105:0/1855245010 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa29c00e070 con 0x7fa2a4105470 2024-04-06T02:42:38.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.281+0000 7fa29bfff700 1 -- 172.21.15.105:0/1855245010 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa29c00b8c0 con 0x7fa2a4105470 2024-04-06T02:42:38.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.281+0000 7fa29bfff700 1 -- 172.21.15.105:0/1855245010 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa29c0044d0 con 0x7fa2a4105470 2024-04-06T02:42:38.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.282+0000 7fa2ab934700 1 -- 172.21.15.105:0/1855245010 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2a4105470 msgr2=0x7fa2a4105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:38.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.282+0000 7fa2ab934700 1 --2- 172.21.15.105:0/1855245010 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2a4105470 0x7fa2a4105890 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa29c009e80 tx=0x7fa29c00b330 comp rx=0 tx=0).stop 2024-04-06T02:42:38.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.282+0000 7fa2ab934700 1 -- 172.21.15.105:0/1855245010 shutdown_connections 2024-04-06T02:42:38.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.282+0000 7fa2ab934700 1 --2- 172.21.15.105:0/1855245010 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2a4105470 0x7fa2a4105890 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:38.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.282+0000 7fa2ab934700 1 -- 172.21.15.105:0/1855245010 >> 172.21.15.105:0/1855245010 conn(0x7fa2a41009f0 msgr2=0x7fa2a4102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:38.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.282+0000 7fa2ab934700 1 -- 172.21.15.105:0/1855245010 shutdown_connections 2024-04-06T02:42:38.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.282+0000 7fa2ab934700 1 -- 172.21.15.105:0/1855245010 wait complete. 2024-04-06T02:42:38.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.283+0000 7fa2ab934700 1 Processor -- start 2024-04-06T02:42:38.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.283+0000 7fa2ab934700 1 -- start start 2024-04-06T02:42:38.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.283+0000 7fa2ab934700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2a4105470 0x7fa2a419a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:38.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.283+0000 7fa2ab934700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a419a650 con 0x7fa2a4105470 2024-04-06T02:42:38.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.283+0000 7fa2a96d0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2a4105470 0x7fa2a419a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:38.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.283+0000 7fa2a96d0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2a4105470 0x7fa2a419a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:45882/0 (socket says 172.21.15.105:45882) 2024-04-06T02:42:38.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.283+0000 7fa2a96d0700 1 -- 172.21.15.105:0/3814301025 learned_addr learned my addr 172.21.15.105:0/3814301025 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:38.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.283+0000 7fa2a96d0700 1 -- 172.21.15.105:0/3814301025 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29c009b30 con 0x7fa2a4105470 2024-04-06T02:42:38.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.283+0000 7fa2a96d0700 1 --2- 172.21.15.105:0/3814301025 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2a4105470 0x7fa2a419a110 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa29c0072a0 tx=0x7fa29c0049f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:38.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.283+0000 7fa29a7fc700 1 -- 172.21.15.105:0/3814301025 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa29c00e040 con 0x7fa2a4105470 2024-04-06T02:42:38.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.284+0000 7fa29a7fc700 1 -- 172.21.15.105:0/3814301025 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa29c00bd80 con 0x7fa2a4105470 2024-04-06T02:42:38.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.284+0000 7fa29a7fc700 1 -- 172.21.15.105:0/3814301025 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa29c0126c0 con 0x7fa2a4105470 2024-04-06T02:42:38.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.284+0000 7fa2ab934700 1 -- 172.21.15.105:0/3814301025 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2a419a850 con 0x7fa2a4105470 2024-04-06T02:42:38.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.284+0000 7fa2ab934700 1 -- 172.21.15.105:0/3814301025 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2a419ad50 con 0x7fa2a4105470 2024-04-06T02:42:38.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.284+0000 7fa29a7fc700 1 -- 172.21.15.105:0/3814301025 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 11) v1 ==== 45190+0+0 (secure 0 0 0) 0x7fa29c019040 con 0x7fa2a4105470 2024-04-06T02:42:38.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.285+0000 7fa29a7fc700 1 --2- 172.21.15.105:0/3814301025 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fa290038260 0x7fa29003a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:38.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.285+0000 7fa29a7fc700 1 -- 172.21.15.105:0/3814301025 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa29c04bdf0 con 0x7fa2a4105470 2024-04-06T02:42:38.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.285+0000 7fa2a8ecf700 1 --2- 172.21.15.105:0/3814301025 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fa290038260 0x7fa29003a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:38.114 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.285+0000 7fa2ab934700 1 -- 172.21.15.105:0/3814301025 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2a40623c0 con 0x7fa2a4105470 2024-04-06T02:42:38.114 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.285+0000 7fa2a8ecf700 1 --2- 172.21.15.105:0/3814301025 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fa290038260 0x7fa29003a720 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa294006fd0 tx=0x7fa294006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:38.114 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.290+0000 7fa29a7fc700 1 -- 172.21.15.105:0/3814301025 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa29c01f020 con 0x7fa2a4105470 2024-04-06T02:42:38.114 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.299+0000 7fa29a7fc700 1 -- 172.21.15.105:0/3814301025 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 12) v1 ==== 45296+0+0 (secure 0 0 0) 0x7fa29c0244f0 con 0x7fa2a4105470 2024-04-06T02:42:38.114 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.492+0000 7fa2ab934700 1 -- 172.21.15.105:0/3814301025 --> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2a4102b00 con 0x7fa290038260 2024-04-06T02:42:38.114 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.572+0000 7fa29a7fc700 1 -- 172.21.15.105:0/3814301025 <== mgr.14162 v2:172.21.15.105:6800/1949998385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fa2a4102b00 con 0x7fa290038260 2024-04-06T02:42:38.114 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.574+0000 7fa2ab934700 1 -- 172.21.15.105:0/3814301025 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fa290038260 msgr2=0x7fa29003a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:38.114 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.574+0000 7fa2ab934700 1 --2- 172.21.15.105:0/3814301025 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fa290038260 0x7fa29003a720 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa294006fd0 tx=0x7fa294006e40 comp rx=0 tx=0).stop 2024-04-06T02:42:38.114 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.574+0000 7fa2ab934700 1 -- 172.21.15.105:0/3814301025 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2a4105470 msgr2=0x7fa2a419a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:38.114 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.574+0000 7fa2ab934700 1 --2- 172.21.15.105:0/3814301025 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2a4105470 0x7fa2a419a110 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa29c0072a0 tx=0x7fa29c0049f0 comp rx=0 tx=0).stop 2024-04-06T02:42:38.114 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.574+0000 7fa2ab934700 1 -- 172.21.15.105:0/3814301025 shutdown_connections 2024-04-06T02:42:38.114 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.574+0000 7fa2ab934700 1 --2- 172.21.15.105:0/3814301025 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fa290038260 0x7fa29003a720 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:38.115 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.574+0000 7fa2ab934700 1 --2- 172.21.15.105:0/3814301025 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2a4105470 0x7fa2a419a110 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:38.115 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.574+0000 7fa2ab934700 1 -- 172.21.15.105:0/3814301025 >> 172.21.15.105:0/3814301025 conn(0x7fa2a41009f0 msgr2=0x7fa2a40740b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:38.115 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.574+0000 7fa2ab934700 1 -- 172.21.15.105:0/3814301025 shutdown_connections 2024-04-06T02:42:38.115 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:37.574+0000 7fa2ab934700 1 -- 172.21.15.105:0/3814301025 wait complete. 2024-04-06T02:42:38.115 INFO:teuthology.orchestra.run.smithi105.stdout:Creating initial admin user... 2024-04-06T02:42:38.513 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:38 smithi105 ceph-mon[29228]: mgrmap e12: smithi105.vqqcpx(active, since 2s) 2024-04-06T02:42:38.514 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:38 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:38.514 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:38 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:38.514 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:38 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:39.333 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:39 smithi105 ceph-mon[29228]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:42:39.333 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:39 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:40.356 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$OONTIjNl/178PKA83f3Rke.XL/lD64MTs/G3JKsRbNEKG5Hg/eMnG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1712371359, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-06T02:42:40.357 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.556+0000 7fe7cd488700 1 Processor -- start 2024-04-06T02:42:40.357 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.556+0000 7fe7cd488700 1 -- start start 2024-04-06T02:42:40.357 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.556+0000 7fe7cd488700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7c8107040 0x7fe7c8109430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:40.357 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.556+0000 7fe7cd488700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c8079e30 con 0x7fe7c8107040 2024-04-06T02:42:40.357 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.557+0000 7fe7c6ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7c8107040 0x7fe7c8109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:40.357 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.557+0000 7fe7c6ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7c8107040 0x7fe7c8109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:45884/0 (socket says 172.21.15.105:45884) 2024-04-06T02:42:40.357 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.557+0000 7fe7c6ffd700 1 -- 172.21.15.105:0/2219426979 learned_addr learned my addr 172.21.15.105:0/2219426979 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:40.357 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.557+0000 7fe7c6ffd700 1 -- 172.21.15.105:0/2219426979 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c8109970 con 0x7fe7c8107040 2024-04-06T02:42:40.357 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.557+0000 7fe7c6ffd700 1 --2- 172.21.15.105:0/2219426979 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7c8107040 0x7fe7c8109430 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe7b80098a0 tx=0x7fe7b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c72134998d9d6d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:40.358 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.557+0000 7fe7c67fc700 1 -- 172.21.15.105:0/2219426979 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7b8004180 con 0x7fe7c8107040 2024-04-06T02:42:40.358 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.557+0000 7fe7c67fc700 1 -- 172.21.15.105:0/2219426979 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe7b80042e0 con 0x7fe7c8107040 2024-04-06T02:42:40.358 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.557+0000 7fe7c67fc700 1 -- 172.21.15.105:0/2219426979 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7b8011af0 con 0x7fe7c8107040 2024-04-06T02:42:40.358 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.558+0000 7fe7cd488700 1 -- 172.21.15.105:0/2219426979 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7c8107040 msgr2=0x7fe7c8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:40.358 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.558+0000 7fe7cd488700 1 --2- 172.21.15.105:0/2219426979 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7c8107040 0x7fe7c8109430 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe7b80098a0 tx=0x7fe7b8009bb0 comp rx=0 tx=0).stop 2024-04-06T02:42:40.358 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.558+0000 7fe7cd488700 1 -- 172.21.15.105:0/2219426979 shutdown_connections 2024-04-06T02:42:40.358 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.558+0000 7fe7cd488700 1 --2- 172.21.15.105:0/2219426979 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7c8107040 0x7fe7c8109430 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:40.358 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.558+0000 7fe7cd488700 1 -- 172.21.15.105:0/2219426979 >> 172.21.15.105:0/2219426979 conn(0x7fe7c81009f0 msgr2=0x7fe7c8102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:40.358 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.558+0000 7fe7cd488700 1 -- 172.21.15.105:0/2219426979 shutdown_connections 2024-04-06T02:42:40.358 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.558+0000 7fe7cd488700 1 -- 172.21.15.105:0/2219426979 wait complete. 2024-04-06T02:42:40.358 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.558+0000 7fe7cd488700 1 Processor -- start 2024-04-06T02:42:40.358 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.558+0000 7fe7cd488700 1 -- start start 2024-04-06T02:42:40.358 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.559+0000 7fe7cd488700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7c8107040 0x7fe7c81069a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:40.359 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.559+0000 7fe7cd488700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c8106ee0 con 0x7fe7c8107040 2024-04-06T02:42:40.359 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.559+0000 7fe7c6ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7c8107040 0x7fe7c81069a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:40.359 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.559+0000 7fe7c6ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7c8107040 0x7fe7c81069a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:45888/0 (socket says 172.21.15.105:45888) 2024-04-06T02:42:40.359 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.559+0000 7fe7c6ffd700 1 -- 172.21.15.105:0/2767737657 learned_addr learned my addr 172.21.15.105:0/2767737657 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:40.359 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.559+0000 7fe7c6ffd700 1 -- 172.21.15.105:0/2767737657 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b8009550 con 0x7fe7c8107040 2024-04-06T02:42:40.359 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.559+0000 7fe7c6ffd700 1 --2- 172.21.15.105:0/2767737657 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7c8107040 0x7fe7c81069a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe7b8006ba0 tx=0x7fe7b8011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:40.359 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.559+0000 7fe7c4ff9700 1 -- 172.21.15.105:0/2767737657 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7b8003b00 con 0x7fe7c8107040 2024-04-06T02:42:40.359 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.559+0000 7fe7cd488700 1 -- 172.21.15.105:0/2767737657 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7c8104ff0 con 0x7fe7c8107040 2024-04-06T02:42:40.359 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.559+0000 7fe7c4ff9700 1 -- 172.21.15.105:0/2767737657 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe7b801a8e0 con 0x7fe7c8107040 2024-04-06T02:42:40.359 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.559+0000 7fe7c4ff9700 1 -- 172.21.15.105:0/2767737657 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7b80199c0 con 0x7fe7c8107040 2024-04-06T02:42:40.359 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.559+0000 7fe7cd488700 1 -- 172.21.15.105:0/2767737657 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c8105490 con 0x7fe7c8107040 2024-04-06T02:42:40.359 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.560+0000 7fe7c4ff9700 1 -- 172.21.15.105:0/2767737657 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 12) v1 ==== 45296+0+0 (secure 0 0 0) 0x7fe7b8019c80 con 0x7fe7c8107040 2024-04-06T02:42:40.359 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.560+0000 7fe7c4ff9700 1 --2- 172.21.15.105:0/2767737657 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fe7b4038320 0x7fe7b403a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:40.359 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.560+0000 7fe7cd488700 1 -- 172.21.15.105:0/2767737657 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7c810a700 con 0x7fe7c8107040 2024-04-06T02:42:40.360 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.560+0000 7fe7c4ff9700 1 -- 172.21.15.105:0/2767737657 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe7b804c7b0 con 0x7fe7c8107040 2024-04-06T02:42:40.360 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.560+0000 7fe7be7fc700 1 --2- 172.21.15.105:0/2767737657 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fe7b4038320 0x7fe7b403a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:40.360 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.561+0000 7fe7be7fc700 1 --2- 172.21.15.105:0/2767737657 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fe7b4038320 0x7fe7b403a7e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0006fd0 tx=0x7fe7b0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:40.360 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.565+0000 7fe7c4ff9700 1 -- 172.21.15.105:0/2767737657 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe7b8018530 con 0x7fe7c8107040 2024-04-06T02:42:40.360 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:38.782+0000 7fe7cd488700 1 -- 172.21.15.105:0/2767737657 --> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fe7c80621e0 con 0x7fe7b4038320 2024-04-06T02:42:40.360 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:39.005+0000 7fe7c4ff9700 1 -- 172.21.15.105:0/2767737657 <== mgr.14162 v2:172.21.15.105:6800/1949998385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fe7c80621e0 con 0x7fe7b4038320 2024-04-06T02:42:40.360 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:39.007+0000 7fe7cd488700 1 -- 172.21.15.105:0/2767737657 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fe7b4038320 msgr2=0x7fe7b403a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:40.360 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:39.007+0000 7fe7cd488700 1 --2- 172.21.15.105:0/2767737657 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fe7b4038320 0x7fe7b403a7e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0006fd0 tx=0x7fe7b0006e40 comp rx=0 tx=0).stop 2024-04-06T02:42:40.360 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:39.007+0000 7fe7cd488700 1 -- 172.21.15.105:0/2767737657 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7c8107040 msgr2=0x7fe7c81069a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:40.360 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:39.007+0000 7fe7cd488700 1 --2- 172.21.15.105:0/2767737657 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7c8107040 0x7fe7c81069a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe7b8006ba0 tx=0x7fe7b8011b50 comp rx=0 tx=0).stop 2024-04-06T02:42:40.360 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:39.007+0000 7fe7cd488700 1 -- 172.21.15.105:0/2767737657 shutdown_connections 2024-04-06T02:42:40.360 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:39.007+0000 7fe7cd488700 1 --2- 172.21.15.105:0/2767737657 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fe7b4038320 0x7fe7b403a7e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:40.360 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:39.007+0000 7fe7cd488700 1 --2- 172.21.15.105:0/2767737657 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7c8107040 0x7fe7c81069a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:40.360 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:39.007+0000 7fe7cd488700 1 -- 172.21.15.105:0/2767737657 >> 172.21.15.105:0/2767737657 conn(0x7fe7c81009f0 msgr2=0x7fe7c81017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:40.361 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:39.007+0000 7fe7cd488700 1 -- 172.21.15.105:0/2767737657 shutdown_connections 2024-04-06T02:42:40.361 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:39.007+0000 7fe7cd488700 1 -- 172.21.15.105:0/2767737657 wait complete. 2024-04-06T02:42:40.361 INFO:teuthology.orchestra.run.smithi105.stdout:Fetching dashboard port number... 2024-04-06T02:42:40.556 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:40 smithi105 ceph-mon[29228]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:42:41.505 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:41 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/1832383750' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-06T02:42:41.512 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 8443 2024-04-06T02:42:41.512 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.766+0000 7f52fcd27700 1 Processor -- start 2024-04-06T02:42:41.512 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.766+0000 7f52fcd27700 1 -- start start 2024-04-06T02:42:41.513 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.767+0000 7f52fcd27700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52f8106560 0x7f52f8106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:41.513 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.767+0000 7f52fcd27700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f8106f50 con 0x7f52f8106560 2024-04-06T02:42:41.513 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.767+0000 7f52f659c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52f8106560 0x7f52f8106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:41.513 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.767+0000 7f52f659c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52f8106560 0x7f52f8106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60072/0 (socket says 172.21.15.105:60072) 2024-04-06T02:42:41.513 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.767+0000 7f52f659c700 1 -- 172.21.15.105:0/368785827 learned_addr learned my addr 172.21.15.105:0/368785827 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:41.513 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.767+0000 7f52f659c700 1 -- 172.21.15.105:0/368785827 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f8107760 con 0x7f52f8106560 2024-04-06T02:42:41.513 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.767+0000 7f52f659c700 1 --2- 172.21.15.105:0/368785827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52f8106560 0x7f52f8106980 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f52e80098a0 tx=0x7f52e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0d3c8fc7d1517a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:41.513 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.767+0000 7f52f559a700 1 -- 172.21.15.105:0/368785827 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52e8004180 con 0x7f52f8106560 2024-04-06T02:42:41.513 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.767+0000 7f52f559a700 1 -- 172.21.15.105:0/368785827 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f52e80042e0 con 0x7f52f8106560 2024-04-06T02:42:41.513 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.767+0000 7f52f559a700 1 -- 172.21.15.105:0/368785827 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52e8011af0 con 0x7f52f8106560 2024-04-06T02:42:41.513 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.768+0000 7f52fcd27700 1 -- 172.21.15.105:0/368785827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52f8106560 msgr2=0x7f52f8106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:41.513 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.768+0000 7f52fcd27700 1 --2- 172.21.15.105:0/368785827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52f8106560 0x7f52f8106980 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f52e80098a0 tx=0x7f52e8009bb0 comp rx=0 tx=0).stop 2024-04-06T02:42:41.513 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.768+0000 7f52fcd27700 1 -- 172.21.15.105:0/368785827 shutdown_connections 2024-04-06T02:42:41.514 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.768+0000 7f52fcd27700 1 --2- 172.21.15.105:0/368785827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52f8106560 0x7f52f8106980 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:41.514 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.768+0000 7f52fcd27700 1 -- 172.21.15.105:0/368785827 >> 172.21.15.105:0/368785827 conn(0x7f52f8101b40 msgr2=0x7f52f8103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:41.514 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.768+0000 7f52fcd27700 1 -- 172.21.15.105:0/368785827 shutdown_connections 2024-04-06T02:42:41.514 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.768+0000 7f52fcd27700 1 -- 172.21.15.105:0/368785827 wait complete. 2024-04-06T02:42:41.514 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.769+0000 7f52fcd27700 1 Processor -- start 2024-04-06T02:42:41.514 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.769+0000 7f52fcd27700 1 -- start start 2024-04-06T02:42:41.514 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.769+0000 7f52fcd27700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52f8106560 0x7f52f819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:41.514 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.769+0000 7f52fcd27700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f819a730 con 0x7f52f8106560 2024-04-06T02:42:41.514 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.769+0000 7f52f659c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52f8106560 0x7f52f819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:41.514 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.769+0000 7f52f659c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52f8106560 0x7f52f819a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60080/0 (socket says 172.21.15.105:60080) 2024-04-06T02:42:41.514 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.769+0000 7f52f659c700 1 -- 172.21.15.105:0/1832383750 learned_addr learned my addr 172.21.15.105:0/1832383750 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:41.514 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.769+0000 7f52f659c700 1 -- 172.21.15.105:0/1832383750 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e8009550 con 0x7f52f8106560 2024-04-06T02:42:41.514 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.769+0000 7f52f659c700 1 --2- 172.21.15.105:0/1832383750 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52f8106560 0x7f52f819a1f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f52e8006ba0 tx=0x7f52e8011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:41.515 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.770+0000 7f52ef7fe700 1 -- 172.21.15.105:0/1832383750 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52e8003b00 con 0x7f52f8106560 2024-04-06T02:42:41.515 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.770+0000 7f52fcd27700 1 -- 172.21.15.105:0/1832383750 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f819a930 con 0x7f52f8106560 2024-04-06T02:42:41.515 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.770+0000 7f52ef7fe700 1 -- 172.21.15.105:0/1832383750 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f52e801a8e0 con 0x7f52f8106560 2024-04-06T02:42:41.515 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.770+0000 7f52ef7fe700 1 -- 172.21.15.105:0/1832383750 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52e8019990 con 0x7f52f8106560 2024-04-06T02:42:41.515 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.770+0000 7f52fcd27700 1 -- 172.21.15.105:0/1832383750 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f819add0 con 0x7f52f8106560 2024-04-06T02:42:41.515 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.770+0000 7f52ef7fe700 1 -- 172.21.15.105:0/1832383750 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 12) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f52e8019c50 con 0x7f52f8106560 2024-04-06T02:42:41.515 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.771+0000 7f52fcd27700 1 -- 172.21.15.105:0/1832383750 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f80623c0 con 0x7f52f8106560 2024-04-06T02:42:41.515 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.771+0000 7f52ef7fe700 1 --2- 172.21.15.105:0/1832383750 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f52e4038320 0x7f52e403a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:41.515 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.771+0000 7f52ef7fe700 1 -- 172.21.15.105:0/1832383750 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f52e804c850 con 0x7f52f8106560 2024-04-06T02:42:41.515 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.771+0000 7f52f5d9b700 1 --2- 172.21.15.105:0/1832383750 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f52e4038320 0x7f52e403a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:41.515 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.771+0000 7f52f5d9b700 1 --2- 172.21.15.105:0/1832383750 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f52e4038320 0x7f52e403a7e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f52e0006fd0 tx=0x7f52e0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:41.515 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.775+0000 7f52ef7fe700 1 -- 172.21.15.105:0/1832383750 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f52e802b700 con 0x7f52f8106560 2024-04-06T02:42:41.515 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.965+0000 7f52fcd27700 1 -- 172.21.15.105:0/1832383750 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f52f819dcd0 con 0x7f52f8106560 2024-04-06T02:42:41.516 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.965+0000 7f52ef7fe700 1 -- 172.21.15.105:0/1832383750 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f52e802b340 con 0x7f52f8106560 2024-04-06T02:42:41.516 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.967+0000 7f52fcd27700 1 -- 172.21.15.105:0/1832383750 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f52e4038320 msgr2=0x7f52e403a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:41.516 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.967+0000 7f52fcd27700 1 --2- 172.21.15.105:0/1832383750 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f52e4038320 0x7f52e403a7e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f52e0006fd0 tx=0x7f52e0006e40 comp rx=0 tx=0).stop 2024-04-06T02:42:41.516 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.967+0000 7f52fcd27700 1 -- 172.21.15.105:0/1832383750 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52f8106560 msgr2=0x7f52f819a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:41.516 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.967+0000 7f52fcd27700 1 --2- 172.21.15.105:0/1832383750 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52f8106560 0x7f52f819a1f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f52e8006ba0 tx=0x7f52e8011b50 comp rx=0 tx=0).stop 2024-04-06T02:42:41.516 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.967+0000 7f52fcd27700 1 -- 172.21.15.105:0/1832383750 shutdown_connections 2024-04-06T02:42:41.516 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.967+0000 7f52fcd27700 1 --2- 172.21.15.105:0/1832383750 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f52e4038320 0x7f52e403a7e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:41.516 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.967+0000 7f52fcd27700 1 --2- 172.21.15.105:0/1832383750 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52f8106560 0x7f52f819a1f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:41.516 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.967+0000 7f52fcd27700 1 -- 172.21.15.105:0/1832383750 >> 172.21.15.105:0/1832383750 conn(0x7f52f8101b40 msgr2=0x7f52f8102890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:41.516 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.967+0000 7f52fcd27700 1 -- 172.21.15.105:0/1832383750 shutdown_connections 2024-04-06T02:42:41.516 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:40.967+0000 7f52fcd27700 1 -- 172.21.15.105:0/1832383750 wait complete. 2024-04-06T02:42:41.519 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-06T02:42:41.519 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout disabled 2024-04-06T02:42:41.525 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-06T02:42:41.525 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout inactive 2024-04-06T02:42:41.525 INFO:teuthology.orchestra.run.smithi105.stdout:firewalld.service is not enabled 2024-04-06T02:42:41.526 INFO:teuthology.orchestra.run.smithi105.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-06T02:42:41.527 INFO:teuthology.orchestra.run.smithi105.stdout:Ceph Dashboard is now available at: 2024-04-06T02:42:41.527 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:42:41.527 INFO:teuthology.orchestra.run.smithi105.stdout: URL: https://smithi105.front.sepia.ceph.com:8443/ 2024-04-06T02:42:41.527 INFO:teuthology.orchestra.run.smithi105.stdout: User: admin 2024-04-06T02:42:41.527 INFO:teuthology.orchestra.run.smithi105.stdout: Password: 8xvzjx7tu3 2024-04-06T02:42:41.528 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:42:41.528 INFO:teuthology.orchestra.run.smithi105.stdout:Saving cluster configuration to /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/config directory 2024-04-06T02:42:41.528 INFO:teuthology.orchestra.run.smithi105.stdout:Enabling autotune for osd_memory_target 2024-04-06T02:42:42.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:42 smithi105 ceph-mon[29228]: mgrmap e13: smithi105.vqqcpx(active, since 6s) 2024-04-06T02:42:42.803 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.957+0000 7f9235ea8700 1 Processor -- start 2024-04-06T02:42:42.803 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.957+0000 7f9235ea8700 1 -- start start 2024-04-06T02:42:42.803 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.957+0000 7f9235ea8700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9230106580 0x7f92301069a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:42.803 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.957+0000 7f9235ea8700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9230106f70 con 0x7f9230106580 2024-04-06T02:42:42.803 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.957+0000 7f922f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9230106580 0x7f92301069a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:42.803 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.957+0000 7f922f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9230106580 0x7f92301069a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60096/0 (socket says 172.21.15.105:60096) 2024-04-06T02:42:42.803 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.957+0000 7f922f7fe700 1 -- 172.21.15.105:0/2335864380 learned_addr learned my addr 172.21.15.105:0/2335864380 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:42.803 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.958+0000 7f922f7fe700 1 -- 172.21.15.105:0/2335864380 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9230107780 con 0x7f9230106580 2024-04-06T02:42:42.803 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.958+0000 7f922f7fe700 1 --2- 172.21.15.105:0/2335864380 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9230106580 0x7f92301069a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f92200098a0 tx=0x7f9220009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3816646c2bdd49b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:42.803 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.958+0000 7f922e7fc700 1 -- 172.21.15.105:0/2335864380 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9220004180 con 0x7f9230106580 2024-04-06T02:42:42.803 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.958+0000 7f922e7fc700 1 -- 172.21.15.105:0/2335864380 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f92200042e0 con 0x7f9230106580 2024-04-06T02:42:42.804 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.958+0000 7f922e7fc700 1 -- 172.21.15.105:0/2335864380 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9220011af0 con 0x7f9230106580 2024-04-06T02:42:42.804 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.958+0000 7f9235ea8700 1 -- 172.21.15.105:0/2335864380 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9230106580 msgr2=0x7f92301069a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:42.804 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.958+0000 7f9235ea8700 1 --2- 172.21.15.105:0/2335864380 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9230106580 0x7f92301069a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f92200098a0 tx=0x7f9220009bb0 comp rx=0 tx=0).stop 2024-04-06T02:42:42.804 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.959+0000 7f9235ea8700 1 -- 172.21.15.105:0/2335864380 shutdown_connections 2024-04-06T02:42:42.804 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.959+0000 7f9235ea8700 1 --2- 172.21.15.105:0/2335864380 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9230106580 0x7f92301069a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:42.804 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.959+0000 7f9235ea8700 1 -- 172.21.15.105:0/2335864380 >> 172.21.15.105:0/2335864380 conn(0x7f9230101b00 msgr2=0x7f9230103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:42.804 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.959+0000 7f9235ea8700 1 -- 172.21.15.105:0/2335864380 shutdown_connections 2024-04-06T02:42:42.804 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.959+0000 7f9235ea8700 1 -- 172.21.15.105:0/2335864380 wait complete. 2024-04-06T02:42:42.804 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.959+0000 7f9235ea8700 1 Processor -- start 2024-04-06T02:42:42.804 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.959+0000 7f9235ea8700 1 -- start start 2024-04-06T02:42:42.804 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.959+0000 7f9235ea8700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9230106580 0x7f923019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:42.804 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.959+0000 7f9235ea8700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923019a7d0 con 0x7f9230106580 2024-04-06T02:42:42.804 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.960+0000 7f922f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9230106580 0x7f923019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:42.804 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.960+0000 7f922f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9230106580 0x7f923019a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60112/0 (socket says 172.21.15.105:60112) 2024-04-06T02:42:42.804 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.960+0000 7f922f7fe700 1 -- 172.21.15.105:0/2458986317 learned_addr learned my addr 172.21.15.105:0/2458986317 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:42.805 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.960+0000 7f922f7fe700 1 -- 172.21.15.105:0/2458986317 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9220009550 con 0x7f9230106580 2024-04-06T02:42:42.805 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.960+0000 7f922f7fe700 1 --2- 172.21.15.105:0/2458986317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9230106580 0x7f923019a290 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9220011670 tx=0x7f9220011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:42.805 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.960+0000 7f922cff9700 1 -- 172.21.15.105:0/2458986317 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92200039f0 con 0x7f9230106580 2024-04-06T02:42:42.805 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.960+0000 7f9235ea8700 1 -- 172.21.15.105:0/2458986317 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f923019a9d0 con 0x7f9230106580 2024-04-06T02:42:42.805 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.960+0000 7f922cff9700 1 -- 172.21.15.105:0/2458986317 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9220003b50 con 0x7f9230106580 2024-04-06T02:42:42.805 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.960+0000 7f922cff9700 1 -- 172.21.15.105:0/2458986317 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92200247c0 con 0x7f9230106580 2024-04-06T02:42:42.805 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.960+0000 7f9235ea8700 1 -- 172.21.15.105:0/2458986317 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f923019ae70 con 0x7f9230106580 2024-04-06T02:42:42.805 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.961+0000 7f922cff9700 1 -- 172.21.15.105:0/2458986317 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f922001a430 con 0x7f9230106580 2024-04-06T02:42:42.805 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.961+0000 7f922cff9700 1 --2- 172.21.15.105:0/2458986317 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f9218038410 0x7f921803a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:42.805 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.961+0000 7f9235ea8700 1 -- 172.21.15.105:0/2458986317 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92300623c0 con 0x7f9230106580 2024-04-06T02:42:42.805 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.961+0000 7f922cff9700 1 -- 172.21.15.105:0/2458986317 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f922004c5a0 con 0x7f9230106580 2024-04-06T02:42:42.805 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.961+0000 7f922effd700 1 --2- 172.21.15.105:0/2458986317 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f9218038410 0x7f921803a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:42.805 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.962+0000 7f922effd700 1 --2- 172.21.15.105:0/2458986317 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f9218038410 0x7f921803a8d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9224006fd0 tx=0x7f9224006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:42.805 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:41.966+0000 7f922cff9700 1 -- 172.21.15.105:0/2458986317 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f9220018b40 con 0x7f9230106580 2024-04-06T02:42:42.806 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:42.154+0000 7f9235ea8700 1 -- 172.21.15.105:0/2458986317 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f923010a980 con 0x7f9230106580 2024-04-06T02:42:42.806 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:42.154+0000 7f922cff9700 1 -- 172.21.15.105:0/2458986317 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f922000f2e0 con 0x7f9230106580 2024-04-06T02:42:42.806 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:42.156+0000 7f9235ea8700 1 -- 172.21.15.105:0/2458986317 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f9218038410 msgr2=0x7f921803a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:42.806 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:42.156+0000 7f9235ea8700 1 --2- 172.21.15.105:0/2458986317 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f9218038410 0x7f921803a8d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9224006fd0 tx=0x7f9224006e40 comp rx=0 tx=0).stop 2024-04-06T02:42:42.806 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:42.156+0000 7f9235ea8700 1 -- 172.21.15.105:0/2458986317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9230106580 msgr2=0x7f923019a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:42.806 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:42.156+0000 7f9235ea8700 1 --2- 172.21.15.105:0/2458986317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9230106580 0x7f923019a290 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9220011670 tx=0x7f9220011b50 comp rx=0 tx=0).stop 2024-04-06T02:42:42.806 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:42.156+0000 7f9235ea8700 1 -- 172.21.15.105:0/2458986317 shutdown_connections 2024-04-06T02:42:42.806 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:42.156+0000 7f9235ea8700 1 --2- 172.21.15.105:0/2458986317 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f9218038410 0x7f921803a8d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:42.806 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:42.156+0000 7f9235ea8700 1 --2- 172.21.15.105:0/2458986317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9230106580 0x7f923019a290 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:42.806 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:42.156+0000 7f9235ea8700 1 -- 172.21.15.105:0/2458986317 >> 172.21.15.105:0/2458986317 conn(0x7f9230101b00 msgr2=0x7f92301031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:42.806 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:42.156+0000 7f9235ea8700 1 -- 172.21.15.105:0/2458986317 shutdown_connections 2024-04-06T02:42:42.806 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:42.156+0000 7f9235ea8700 1 -- 172.21.15.105:0/2458986317 wait complete. 2024-04-06T02:42:44.068 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.222+0000 7f51bcde9700 1 Processor -- start 2024-04-06T02:42:44.068 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.222+0000 7f51bcde9700 1 -- start start 2024-04-06T02:42:44.068 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.222+0000 7f51bcde9700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51b8106560 0x7f51b8106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:44.068 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.222+0000 7f51bcde9700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51b8106f50 con 0x7f51b8106560 2024-04-06T02:42:44.068 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.222+0000 7f51b659c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51b8106560 0x7f51b8106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:44.068 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.223+0000 7f51b659c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51b8106560 0x7f51b8106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60120/0 (socket says 172.21.15.105:60120) 2024-04-06T02:42:44.069 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.223+0000 7f51b659c700 1 -- 172.21.15.105:0/2047418153 learned_addr learned my addr 172.21.15.105:0/2047418153 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:44.069 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.223+0000 7f51b659c700 1 -- 172.21.15.105:0/2047418153 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b8107760 con 0x7f51b8106560 2024-04-06T02:42:44.069 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.223+0000 7f51b659c700 1 --2- 172.21.15.105:0/2047418153 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51b8106560 0x7f51b8106980 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f51a80098a0 tx=0x7f51a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d308397db5cf8161 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:44.069 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.223+0000 7f51b559a700 1 -- 172.21.15.105:0/2047418153 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51a8004180 con 0x7f51b8106560 2024-04-06T02:42:44.069 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.223+0000 7f51b559a700 1 -- 172.21.15.105:0/2047418153 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f51a80042e0 con 0x7f51b8106560 2024-04-06T02:42:44.069 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.223+0000 7f51b559a700 1 -- 172.21.15.105:0/2047418153 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51a8011af0 con 0x7f51b8106560 2024-04-06T02:42:44.069 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.224+0000 7f51bcde9700 1 -- 172.21.15.105:0/2047418153 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51b8106560 msgr2=0x7f51b8106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:44.069 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.224+0000 7f51bcde9700 1 --2- 172.21.15.105:0/2047418153 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51b8106560 0x7f51b8106980 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f51a80098a0 tx=0x7f51a8009bb0 comp rx=0 tx=0).stop 2024-04-06T02:42:44.069 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.224+0000 7f51bcde9700 1 -- 172.21.15.105:0/2047418153 shutdown_connections 2024-04-06T02:42:44.069 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.224+0000 7f51bcde9700 1 --2- 172.21.15.105:0/2047418153 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51b8106560 0x7f51b8106980 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:44.069 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.224+0000 7f51bcde9700 1 -- 172.21.15.105:0/2047418153 >> 172.21.15.105:0/2047418153 conn(0x7f51b8101b40 msgr2=0x7f51b8103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:44.069 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.224+0000 7f51bcde9700 1 -- 172.21.15.105:0/2047418153 shutdown_connections 2024-04-06T02:42:44.069 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.224+0000 7f51bcde9700 1 -- 172.21.15.105:0/2047418153 wait complete. 2024-04-06T02:42:44.069 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.224+0000 7f51bcde9700 1 Processor -- start 2024-04-06T02:42:44.070 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.225+0000 7f51bcde9700 1 -- start start 2024-04-06T02:42:44.070 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.225+0000 7f51bcde9700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51b8106560 0x7f51b819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:44.070 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.225+0000 7f51bcde9700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51b819a730 con 0x7f51b8106560 2024-04-06T02:42:44.070 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.225+0000 7f51b659c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51b8106560 0x7f51b819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:44.070 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.225+0000 7f51b659c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51b8106560 0x7f51b819a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60130/0 (socket says 172.21.15.105:60130) 2024-04-06T02:42:44.070 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.225+0000 7f51b659c700 1 -- 172.21.15.105:0/1785514557 learned_addr learned my addr 172.21.15.105:0/1785514557 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:44.070 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.225+0000 7f51b659c700 1 -- 172.21.15.105:0/1785514557 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51a8009550 con 0x7f51b8106560 2024-04-06T02:42:44.070 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.225+0000 7f51b659c700 1 --2- 172.21.15.105:0/1785514557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51b8106560 0x7f51b819a1f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f51a8006ba0 tx=0x7f51a8011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:44.070 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.225+0000 7f51a77fe700 1 -- 172.21.15.105:0/1785514557 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51a8003b00 con 0x7f51b8106560 2024-04-06T02:42:44.070 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.225+0000 7f51a77fe700 1 -- 172.21.15.105:0/1785514557 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f51a801a8e0 con 0x7f51b8106560 2024-04-06T02:42:44.070 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.225+0000 7f51bcde9700 1 -- 172.21.15.105:0/1785514557 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51b819a930 con 0x7f51b8106560 2024-04-06T02:42:44.070 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.225+0000 7f51bcde9700 1 -- 172.21.15.105:0/1785514557 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51b819add0 con 0x7f51b8106560 2024-04-06T02:42:44.070 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.225+0000 7f51a77fe700 1 -- 172.21.15.105:0/1785514557 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51a8019990 con 0x7f51b8106560 2024-04-06T02:42:44.070 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.226+0000 7f51a77fe700 1 -- 172.21.15.105:0/1785514557 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f51a8019c50 con 0x7f51b8106560 2024-04-06T02:42:44.071 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.226+0000 7f51bcde9700 1 -- 172.21.15.105:0/1785514557 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51b80623c0 con 0x7f51b8106560 2024-04-06T02:42:44.071 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.226+0000 7f51a77fe700 1 --2- 172.21.15.105:0/1785514557 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f51a00383c0 0x7f51a003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:44.071 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.227+0000 7f51a77fe700 1 -- 172.21.15.105:0/1785514557 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f51a804ca00 con 0x7f51b8106560 2024-04-06T02:42:44.071 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.227+0000 7f51b5d9b700 1 --2- 172.21.15.105:0/1785514557 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f51a00383c0 0x7f51a003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:44.071 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.227+0000 7f51b5d9b700 1 --2- 172.21.15.105:0/1785514557 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f51a00383c0 0x7f51a003a880 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f51ac006fd0 tx=0x7f51ac006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:44.071 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.231+0000 7f51a77fe700 1 -- 172.21.15.105:0/1785514557 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f51a802d440 con 0x7f51b8106560 2024-04-06T02:42:44.071 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.528+0000 7f51bcde9700 1 -- 172.21.15.105:0/1785514557 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f51b819dcd0 con 0x7f51b8106560 2024-04-06T02:42:44.071 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.530+0000 7f51a77fe700 1 -- 172.21.15.105:0/1785514557 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v34)=0 set mgr/dashboard/cluster/status v34) v1 ==== 153+0+0 (secure 0 0 0) 0x7f51a802b340 con 0x7f51b8106560 2024-04-06T02:42:44.071 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.532+0000 7f51bcde9700 1 -- 172.21.15.105:0/1785514557 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f51a00383c0 msgr2=0x7f51a003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:44.071 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.532+0000 7f51bcde9700 1 --2- 172.21.15.105:0/1785514557 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f51a00383c0 0x7f51a003a880 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f51ac006fd0 tx=0x7f51ac006e40 comp rx=0 tx=0).stop 2024-04-06T02:42:44.071 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.532+0000 7f51bcde9700 1 -- 172.21.15.105:0/1785514557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51b8106560 msgr2=0x7f51b819a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:44.071 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.532+0000 7f51bcde9700 1 --2- 172.21.15.105:0/1785514557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51b8106560 0x7f51b819a1f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f51a8006ba0 tx=0x7f51a8011b50 comp rx=0 tx=0).stop 2024-04-06T02:42:44.071 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.532+0000 7f51bcde9700 1 -- 172.21.15.105:0/1785514557 shutdown_connections 2024-04-06T02:42:44.071 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.532+0000 7f51bcde9700 1 --2- 172.21.15.105:0/1785514557 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f51a00383c0 0x7f51a003a880 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:44.072 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.532+0000 7f51bcde9700 1 --2- 172.21.15.105:0/1785514557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51b8106560 0x7f51b819a1f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:44.072 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.532+0000 7f51bcde9700 1 -- 172.21.15.105:0/1785514557 >> 172.21.15.105:0/1785514557 conn(0x7f51b8101b40 msgr2=0x7f51b8102890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:44.072 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.532+0000 7f51bcde9700 1 -- 172.21.15.105:0/1785514557 shutdown_connections 2024-04-06T02:42:44.072 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-06T02:42:43.532+0000 7f51bcde9700 1 -- 172.21.15.105:0/1785514557 wait complete. 2024-04-06T02:42:44.072 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-06T02:42:44.072 INFO:teuthology.orchestra.run.smithi105.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-06T02:42:44.072 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:42:44.072 INFO:teuthology.orchestra.run.smithi105.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-06T02:42:44.072 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:42:44.072 INFO:teuthology.orchestra.run.smithi105.stdout:Or, if you are only running a single cluster on this host: 2024-04-06T02:42:44.072 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:42:44.072 INFO:teuthology.orchestra.run.smithi105.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-06T02:42:44.072 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:42:44.072 INFO:teuthology.orchestra.run.smithi105.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-06T02:42:44.072 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:42:44.073 INFO:teuthology.orchestra.run.smithi105.stdout: ceph telemetry on 2024-04-06T02:42:44.073 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:42:44.073 INFO:teuthology.orchestra.run.smithi105.stdout:For more information see: 2024-04-06T02:42:44.073 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:42:44.073 INFO:teuthology.orchestra.run.smithi105.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-06T02:42:44.073 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:42:44.073 INFO:teuthology.orchestra.run.smithi105.stdout:Bootstrap complete. 2024-04-06T02:42:44.101 INFO:tasks.cephadm:Fetching config... 2024-04-06T02:42:44.102 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-06T02:42:44.102 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-06T02:42:44.158 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-06T02:42:44.158 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-06T02:42:44.158 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-06T02:42:44.213 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-06T02:42:44.213 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-06T02:42:44.213 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/keyring of=/dev/stdout 2024-04-06T02:42:44.281 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-06T02:42:44.281 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-06T02:42:44.281 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-06T02:42:44.335 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-06T02:42:44.336 DEBUG:teuthology.orchestra.run.smithi105:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMS5W7QPe7kQnHN+Ai3JPQqf38tfmBhbG/l2unsLPQjKI593vabhr3YyySPU36wF7C3aq0KFas0BCZvPXHpZ41WO0SEjRPhLLNqQbopbFXZr4n94AMChMlg1ovkVKQdIV6fjSMr0foGoS1rRzFoYWD3wJGRDuw5WrQQTanjE6Lbz9LWeniCFbT8TUXIUpDGURxSDLCmr0rYgwTOC+pQizeaEe4sVrhARzlMI9J7Nu5oQ22iFz0yDhkPLhDR70kr2I7lwYZ7FmdHhUynF/XsttuagDwL1bmdHAIw5Y7jljBJvUyZzgXPkt1lMkLaEZuL1qfgn/1fKHawaMuCeY8K2GCyzW7mv3wtJhUrw/jMRuQR5x8mRmXQmzT1R06faa/bwCsQH7PDXak6yMRrGXzom0Ja80miJI9WD+Cr/3ne9cmypy80shjIX2KgvTbRuDfwUjqs1Q0D2+ebArUebGWacnCACg27WXnDbB7ONwaRn8l+W6Omdhti9ZAOSfOly2oV2c= ceph-130df578-f3bf-11ee-b649-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-06T02:42:44.415 INFO:teuthology.orchestra.run.smithi105.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMS5W7QPe7kQnHN+Ai3JPQqf38tfmBhbG/l2unsLPQjKI593vabhr3YyySPU36wF7C3aq0KFas0BCZvPXHpZ41WO0SEjRPhLLNqQbopbFXZr4n94AMChMlg1ovkVKQdIV6fjSMr0foGoS1rRzFoYWD3wJGRDuw5WrQQTanjE6Lbz9LWeniCFbT8TUXIUpDGURxSDLCmr0rYgwTOC+pQizeaEe4sVrhARzlMI9J7Nu5oQ22iFz0yDhkPLhDR70kr2I7lwYZ7FmdHhUynF/XsttuagDwL1bmdHAIw5Y7jljBJvUyZzgXPkt1lMkLaEZuL1qfgn/1fKHawaMuCeY8K2GCyzW7mv3wtJhUrw/jMRuQR5x8mRmXQmzT1R06faa/bwCsQH7PDXak6yMRrGXzom0Ja80miJI9WD+Cr/3ne9cmypy80shjIX2KgvTbRuDfwUjqs1Q0D2+ebArUebGWacnCACg27WXnDbB7ONwaRn8l+W6Omdhti9ZAOSfOly2oV2c= ceph-130df578-f3bf-11ee-b649-cb9ed24678a4 2024-04-06T02:42:44.429 DEBUG:teuthology.orchestra.run.smithi184:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMS5W7QPe7kQnHN+Ai3JPQqf38tfmBhbG/l2unsLPQjKI593vabhr3YyySPU36wF7C3aq0KFas0BCZvPXHpZ41WO0SEjRPhLLNqQbopbFXZr4n94AMChMlg1ovkVKQdIV6fjSMr0foGoS1rRzFoYWD3wJGRDuw5WrQQTanjE6Lbz9LWeniCFbT8TUXIUpDGURxSDLCmr0rYgwTOC+pQizeaEe4sVrhARzlMI9J7Nu5oQ22iFz0yDhkPLhDR70kr2I7lwYZ7FmdHhUynF/XsttuagDwL1bmdHAIw5Y7jljBJvUyZzgXPkt1lMkLaEZuL1qfgn/1fKHawaMuCeY8K2GCyzW7mv3wtJhUrw/jMRuQR5x8mRmXQmzT1R06faa/bwCsQH7PDXak6yMRrGXzom0Ja80miJI9WD+Cr/3ne9cmypy80shjIX2KgvTbRuDfwUjqs1Q0D2+ebArUebGWacnCACg27WXnDbB7ONwaRn8l+W6Omdhti9ZAOSfOly2oV2c= ceph-130df578-f3bf-11ee-b649-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-06T02:42:44.478 INFO:teuthology.orchestra.run.smithi184.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMS5W7QPe7kQnHN+Ai3JPQqf38tfmBhbG/l2unsLPQjKI593vabhr3YyySPU36wF7C3aq0KFas0BCZvPXHpZ41WO0SEjRPhLLNqQbopbFXZr4n94AMChMlg1ovkVKQdIV6fjSMr0foGoS1rRzFoYWD3wJGRDuw5WrQQTanjE6Lbz9LWeniCFbT8TUXIUpDGURxSDLCmr0rYgwTOC+pQizeaEe4sVrhARzlMI9J7Nu5oQ22iFz0yDhkPLhDR70kr2I7lwYZ7FmdHhUynF/XsttuagDwL1bmdHAIw5Y7jljBJvUyZzgXPkt1lMkLaEZuL1qfgn/1fKHawaMuCeY8K2GCyzW7mv3wtJhUrw/jMRuQR5x8mRmXQmzT1R06faa/bwCsQH7PDXak6yMRrGXzom0Ja80miJI9WD+Cr/3ne9cmypy80shjIX2KgvTbRuDfwUjqs1Q0D2+ebArUebGWacnCACg27WXnDbB7ONwaRn8l+W6Omdhti9ZAOSfOly2oV2c= ceph-130df578-f3bf-11ee-b649-cb9ed24678a4 2024-04-06T02:42:44.493 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-06T02:42:44.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:44 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/1785514557' entity='client.admin' 2024-04-06T02:42:44.697 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:42:47.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.132+0000 7f4810eb4700 1 -- 172.21.15.105:0/3122740792 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f480c101330 msgr2=0x7f480c103720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:47.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.132+0000 7f4810eb4700 1 --2- 172.21.15.105:0/3122740792 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f480c101330 0x7f480c103720 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f47fc009910 tx=0x7f47fc009c20 comp rx=0 tx=0).stop 2024-04-06T02:42:47.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.132+0000 7f4810eb4700 1 -- 172.21.15.105:0/3122740792 shutdown_connections 2024-04-06T02:42:47.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.132+0000 7f4810eb4700 1 --2- 172.21.15.105:0/3122740792 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f480c101330 0x7f480c103720 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:47.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.132+0000 7f4810eb4700 1 -- 172.21.15.105:0/3122740792 >> 172.21.15.105:0/3122740792 conn(0x7f480c0fad20 msgr2=0x7f480c0fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:47.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.132+0000 7f4810eb4700 1 -- 172.21.15.105:0/3122740792 shutdown_connections 2024-04-06T02:42:47.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.132+0000 7f4810eb4700 1 -- 172.21.15.105:0/3122740792 wait complete. 2024-04-06T02:42:47.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.132+0000 7f4810eb4700 1 Processor -- start 2024-04-06T02:42:47.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.133+0000 7f4810eb4700 1 -- start start 2024-04-06T02:42:47.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.133+0000 7f4810eb4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f480c101330 0x7f480c195930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:47.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.133+0000 7f4810eb4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f480c195e70 con 0x7f480c101330 2024-04-06T02:42:47.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.133+0000 7f480a59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f480c101330 0x7f480c195930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:47.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.133+0000 7f480a59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f480c101330 0x7f480c195930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60140/0 (socket says 172.21.15.105:60140) 2024-04-06T02:42:47.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.133+0000 7f480a59c700 1 -- 172.21.15.105:0/2059597237 learned_addr learned my addr 172.21.15.105:0/2059597237 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:47.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.133+0000 7f480a59c700 1 -- 172.21.15.105:0/2059597237 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47fc0095f0 con 0x7f480c101330 2024-04-06T02:42:47.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.133+0000 7f480a59c700 1 --2- 172.21.15.105:0/2059597237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f480c101330 0x7f480c195930 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f47fc0092e0 tx=0x7f47fc009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:47.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.133+0000 7f4803fff700 1 -- 172.21.15.105:0/2059597237 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47fc01d070 con 0x7f480c101330 2024-04-06T02:42:47.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.133+0000 7f4810eb4700 1 -- 172.21.15.105:0/2059597237 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f480c196070 con 0x7f480c101330 2024-04-06T02:42:47.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.133+0000 7f4803fff700 1 -- 172.21.15.105:0/2059597237 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f47fc00be10 con 0x7f480c101330 2024-04-06T02:42:47.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.133+0000 7f4803fff700 1 -- 172.21.15.105:0/2059597237 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47fc00f460 con 0x7f480c101330 2024-04-06T02:42:47.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.133+0000 7f4810eb4700 1 -- 172.21.15.105:0/2059597237 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f480c196510 con 0x7f480c101330 2024-04-06T02:42:47.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.134+0000 7f4803fff700 1 -- 172.21.15.105:0/2059597237 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f47fc00f6d0 con 0x7f480c101330 2024-04-06T02:42:47.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.134+0000 7f4803fff700 1 --2- 172.21.15.105:0/2059597237 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f47f80383c0 0x7f47f803a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:47.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.134+0000 7f4803fff700 1 -- 172.21.15.105:0/2059597237 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f47fc022470 con 0x7f480c101330 2024-04-06T02:42:47.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.134+0000 7f4810eb4700 1 -- 172.21.15.105:0/2059597237 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f480c18f4c0 con 0x7f480c101330 2024-04-06T02:42:47.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.135+0000 7f4801d9b700 1 --2- 172.21.15.105:0/2059597237 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f47f80383c0 0x7f47f803a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:47.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.135+0000 7f4801d9b700 1 --2- 172.21.15.105:0/2059597237 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f47f80383c0 0x7f47f803a880 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f47f4006fd0 tx=0x7f47f4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:47.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.139+0000 7f4803fff700 1 -- 172.21.15.105:0/2059597237 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f47fc027070 con 0x7f480c101330 2024-04-06T02:42:47.315 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.331+0000 7f4810eb4700 1 -- 172.21.15.105:0/2059597237 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f480c02cf20 con 0x7f480c101330 2024-04-06T02:42:47.320 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.336+0000 7f4803fff700 1 -- 172.21.15.105:0/2059597237 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f47fc02c440 con 0x7f480c101330 2024-04-06T02:42:47.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.338+0000 7f4810eb4700 1 -- 172.21.15.105:0/2059597237 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f47f80383c0 msgr2=0x7f47f803a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:47.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.338+0000 7f4810eb4700 1 --2- 172.21.15.105:0/2059597237 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f47f80383c0 0x7f47f803a880 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f47f4006fd0 tx=0x7f47f4006e40 comp rx=0 tx=0).stop 2024-04-06T02:42:47.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.338+0000 7f4810eb4700 1 -- 172.21.15.105:0/2059597237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f480c101330 msgr2=0x7f480c195930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:47.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.338+0000 7f4810eb4700 1 --2- 172.21.15.105:0/2059597237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f480c101330 0x7f480c195930 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f47fc0092e0 tx=0x7f47fc009e60 comp rx=0 tx=0).stop 2024-04-06T02:42:47.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.338+0000 7f4810eb4700 1 -- 172.21.15.105:0/2059597237 shutdown_connections 2024-04-06T02:42:47.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.338+0000 7f4810eb4700 1 --2- 172.21.15.105:0/2059597237 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f47f80383c0 0x7f47f803a880 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:47.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.338+0000 7f4810eb4700 1 --2- 172.21.15.105:0/2059597237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f480c101330 0x7f480c195930 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:47.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.338+0000 7f4810eb4700 1 -- 172.21.15.105:0/2059597237 >> 172.21.15.105:0/2059597237 conn(0x7f480c0fad20 msgr2=0x7f480c0fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:47.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.338+0000 7f4810eb4700 1 -- 172.21.15.105:0/2059597237 shutdown_connections 2024-04-06T02:42:47.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:47.338+0000 7f4810eb4700 1 -- 172.21.15.105:0/2059597237 wait complete. 2024-04-06T02:42:48.006 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-06T02:42:48.007 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-06T02:42:48.563 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:48 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/2059597237' entity='client.admin' 2024-04-06T02:42:48.638 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:42:49.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:49 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:49.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:49 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:49.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:49 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config rm", "who": "osd/host:smithi105", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:42:49.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:49 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:49.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:49 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi105", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-06T02:42:49.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:49 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi105", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-06T02:42:49.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:49 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:42:49.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:49 smithi105 ceph-mon[29228]: Deploying daemon ceph-exporter.smithi105 on smithi105 2024-04-06T02:42:50.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.175+0000 7fd22ca69700 1 -- 172.21.15.105:0/4106556881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2281008a0 msgr2=0x7fd228100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:50.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.175+0000 7fd22ca69700 1 --2- 172.21.15.105:0/4106556881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2281008a0 0x7fd228100cc0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd218009910 tx=0x7fd218009c20 comp rx=0 tx=0).stop 2024-04-06T02:42:50.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.175+0000 7fd22ca69700 1 -- 172.21.15.105:0/4106556881 shutdown_connections 2024-04-06T02:42:50.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.175+0000 7fd22ca69700 1 --2- 172.21.15.105:0/4106556881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2281008a0 0x7fd228100cc0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:50.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.175+0000 7fd22ca69700 1 -- 172.21.15.105:0/4106556881 >> 172.21.15.105:0/4106556881 conn(0x7fd2280fbe60 msgr2=0x7fd2280fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:50.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.175+0000 7fd22ca69700 1 -- 172.21.15.105:0/4106556881 shutdown_connections 2024-04-06T02:42:50.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.175+0000 7fd22ca69700 1 -- 172.21.15.105:0/4106556881 wait complete. 2024-04-06T02:42:50.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.176+0000 7fd22ca69700 1 Processor -- start 2024-04-06T02:42:50.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.176+0000 7fd22ca69700 1 -- start start 2024-04-06T02:42:50.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.176+0000 7fd22ca69700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2281008a0 0x7fd228197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:50.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.176+0000 7fd22ca69700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd228198110 con 0x7fd2281008a0 2024-04-06T02:42:50.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.176+0000 7fd22659c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2281008a0 0x7fd228197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:50.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.176+0000 7fd22659c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2281008a0 0x7fd228197bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60158/0 (socket says 172.21.15.105:60158) 2024-04-06T02:42:50.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.176+0000 7fd22659c700 1 -- 172.21.15.105:0/4012738950 learned_addr learned my addr 172.21.15.105:0/4012738950 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:50.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.177+0000 7fd22659c700 1 -- 172.21.15.105:0/4012738950 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2180095f0 con 0x7fd2281008a0 2024-04-06T02:42:50.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.177+0000 7fd22659c700 1 --2- 172.21.15.105:0/4012738950 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2281008a0 0x7fd228197bd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd218005b40 tx=0x7fd218004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:50.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.177+0000 7fd2177fe700 1 -- 172.21.15.105:0/4012738950 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd21801d070 con 0x7fd2281008a0 2024-04-06T02:42:50.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.177+0000 7fd22ca69700 1 -- 172.21.15.105:0/4012738950 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd228198310 con 0x7fd2281008a0 2024-04-06T02:42:50.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.177+0000 7fd2177fe700 1 -- 172.21.15.105:0/4012738950 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd21800be10 con 0x7fd2281008a0 2024-04-06T02:42:50.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.177+0000 7fd2177fe700 1 -- 172.21.15.105:0/4012738950 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd21800f460 con 0x7fd2281008a0 2024-04-06T02:42:50.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.177+0000 7fd22ca69700 1 -- 172.21.15.105:0/4012738950 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2281987b0 con 0x7fd2281008a0 2024-04-06T02:42:50.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.178+0000 7fd2177fe700 1 -- 172.21.15.105:0/4012738950 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fd21800f6d0 con 0x7fd2281008a0 2024-04-06T02:42:50.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.178+0000 7fd2177fe700 1 --2- 172.21.15.105:0/4012738950 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fd2100383c0 0x7fd21003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:50.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.178+0000 7fd2177fe700 1 -- 172.21.15.105:0/4012738950 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd21804c530 con 0x7fd2281008a0 2024-04-06T02:42:50.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.178+0000 7fd22ca69700 1 -- 172.21.15.105:0/4012738950 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2281918c0 con 0x7fd2281008a0 2024-04-06T02:42:50.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.178+0000 7fd225d9b700 1 --2- 172.21.15.105:0/4012738950 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fd2100383c0 0x7fd21003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:50.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.178+0000 7fd225d9b700 1 --2- 172.21.15.105:0/4012738950 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fd2100383c0 0x7fd21003a880 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd21c006fd0 tx=0x7fd21c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:50.167 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.183+0000 7fd2177fe700 1 -- 172.21.15.105:0/4012738950 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd218027070 con 0x7fd2281008a0 2024-04-06T02:42:50.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.396+0000 7fd22ca69700 1 -- 172.21.15.105:0/4012738950 --> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fd228061260 con 0x7fd2100383c0 2024-04-06T02:42:50.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.399+0000 7fd2177fe700 1 -- 172.21.15.105:0/4012738950 <== mgr.14162 v2:172.21.15.105:6800/1949998385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd228061260 con 0x7fd2100383c0 2024-04-06T02:42:50.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.401+0000 7fd22ca69700 1 -- 172.21.15.105:0/4012738950 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fd2100383c0 msgr2=0x7fd21003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:50.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.401+0000 7fd22ca69700 1 --2- 172.21.15.105:0/4012738950 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fd2100383c0 0x7fd21003a880 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd21c006fd0 tx=0x7fd21c006e40 comp rx=0 tx=0).stop 2024-04-06T02:42:50.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.401+0000 7fd22ca69700 1 -- 172.21.15.105:0/4012738950 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2281008a0 msgr2=0x7fd228197bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:50.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.402+0000 7fd22ca69700 1 --2- 172.21.15.105:0/4012738950 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2281008a0 0x7fd228197bd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd218005b40 tx=0x7fd218004740 comp rx=0 tx=0).stop 2024-04-06T02:42:50.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.402+0000 7fd22ca69700 1 -- 172.21.15.105:0/4012738950 shutdown_connections 2024-04-06T02:42:50.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.402+0000 7fd22ca69700 1 --2- 172.21.15.105:0/4012738950 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fd2100383c0 0x7fd21003a880 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:50.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.402+0000 7fd22ca69700 1 --2- 172.21.15.105:0/4012738950 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2281008a0 0x7fd228197bd0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:50.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.402+0000 7fd22ca69700 1 -- 172.21.15.105:0/4012738950 >> 172.21.15.105:0/4012738950 conn(0x7fd2280fbe60 msgr2=0x7fd2280fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:50.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.402+0000 7fd22ca69700 1 -- 172.21.15.105:0/4012738950 shutdown_connections 2024-04-06T02:42:50.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:50.402+0000 7fd22ca69700 1 -- 172.21.15.105:0/4012738950 wait complete. 2024-04-06T02:42:50.891 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi184 2024-04-06T02:42:50.891 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-04-06T02:42:50.891 DEBUG:teuthology.orchestra.run.smithi184:> dd of=/etc/ceph/ceph.conf 2024-04-06T02:42:50.909 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-04-06T02:42:50.910 DEBUG:teuthology.orchestra.run.smithi184:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:42:50.967 INFO:tasks.cephadm:Adding host smithi184 to orchestrator... 2024-04-06T02:42:50.967 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph orch host add smithi184 2024-04-06T02:42:51.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:51 smithi105 ceph-mon[29228]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:42:51.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:51 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:51.903 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:42:53.839 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.852+0000 7ff8069b2700 1 -- 172.21.15.105:0/1675462294 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff8000fef30 msgr2=0x7ff8000ff350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:53.839 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.852+0000 7ff8069b2700 1 --2- 172.21.15.105:0/1675462294 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff8000fef30 0x7ff8000ff350 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0009910 tx=0x7ff7f0009c20 comp rx=0 tx=0).stop 2024-04-06T02:42:53.839 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.853+0000 7ff8069b2700 1 -- 172.21.15.105:0/1675462294 shutdown_connections 2024-04-06T02:42:53.839 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.853+0000 7ff8069b2700 1 --2- 172.21.15.105:0/1675462294 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff8000fef30 0x7ff8000ff350 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:53.839 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.853+0000 7ff8069b2700 1 -- 172.21.15.105:0/1675462294 >> 172.21.15.105:0/1675462294 conn(0x7ff8000fa4d0 msgr2=0x7ff8000fc910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:53.839 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.853+0000 7ff8069b2700 1 -- 172.21.15.105:0/1675462294 shutdown_connections 2024-04-06T02:42:53.839 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.853+0000 7ff8069b2700 1 -- 172.21.15.105:0/1675462294 wait complete. 2024-04-06T02:42:53.839 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.853+0000 7ff8069b2700 1 Processor -- start 2024-04-06T02:42:53.839 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.853+0000 7ff8069b2700 1 -- start start 2024-04-06T02:42:53.840 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.854+0000 7ff8069b2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff8000fef30 0x7ff8001959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:53.840 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.854+0000 7ff8069b2700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff800195f00 con 0x7ff8000fef30 2024-04-06T02:42:53.840 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.854+0000 7ff7fffff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff8000fef30 0x7ff8001959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:53.840 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.854+0000 7ff7fffff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff8000fef30 0x7ff8001959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43120/0 (socket says 172.21.15.105:43120) 2024-04-06T02:42:53.840 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.854+0000 7ff7fffff700 1 -- 172.21.15.105:0/1418692767 learned_addr learned my addr 172.21.15.105:0/1418692767 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:53.840 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.854+0000 7ff7fffff700 1 -- 172.21.15.105:0/1418692767 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f00095f0 con 0x7ff8000fef30 2024-04-06T02:42:53.840 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.854+0000 7ff7fffff700 1 --2- 172.21.15.105:0/1418692767 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff8000fef30 0x7ff8001959c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0005b40 tx=0x7ff7f0004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:53.840 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.854+0000 7ff7fd7fa700 1 -- 172.21.15.105:0/1418692767 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7f001d070 con 0x7ff8000fef30 2024-04-06T02:42:53.840 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.854+0000 7ff7fd7fa700 1 -- 172.21.15.105:0/1418692767 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff7f000be10 con 0x7ff8000fef30 2024-04-06T02:42:53.840 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.854+0000 7ff8069b2700 1 -- 172.21.15.105:0/1418692767 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff800196100 con 0x7ff8000fef30 2024-04-06T02:42:53.840 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.854+0000 7ff7fd7fa700 1 -- 172.21.15.105:0/1418692767 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7f000f460 con 0x7ff8000fef30 2024-04-06T02:42:53.840 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.854+0000 7ff8069b2700 1 -- 172.21.15.105:0/1418692767 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8001965a0 con 0x7ff8000fef30 2024-04-06T02:42:53.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.855+0000 7ff7fd7fa700 1 -- 172.21.15.105:0/1418692767 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7ff7f000f5c0 con 0x7ff8000fef30 2024-04-06T02:42:53.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.855+0000 7ff7fd7fa700 1 --2- 172.21.15.105:0/1418692767 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7ff7e8038410 0x7ff7e803a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:53.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.855+0000 7ff8069b2700 1 -- 172.21.15.105:0/1418692767 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff80018fcd0 con 0x7ff8000fef30 2024-04-06T02:42:53.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.855+0000 7ff7fd7fa700 1 -- 172.21.15.105:0/1418692767 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff7f004c3b0 con 0x7ff8000fef30 2024-04-06T02:42:53.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.855+0000 7ff7ff7fe700 1 --2- 172.21.15.105:0/1418692767 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7ff7e8038410 0x7ff7e803a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:53.842 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.856+0000 7ff7ff7fe700 1 --2- 172.21.15.105:0/1418692767 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7ff7e8038410 0x7ff7e803a8d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4006fd0 tx=0x7ff7f4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:53.844 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:53.860+0000 7ff7fd7fa700 1 -- 172.21.15.105:0/1418692767 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff7f0027070 con 0x7ff8000fef30 2024-04-06T02:42:54.042 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:54.058+0000 7ff8069b2700 1 -- 172.21.15.105:0/1418692767 --> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi184", "target": ["mon-mgr", ""]}) v1 -- 0x7ff8000611d0 con 0x7ff7e8038410 2024-04-06T02:42:55.403 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:55 smithi105 ceph-mon[29228]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi184", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:42:55.403 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:55 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:55.403 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:55 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:55.403 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:55 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:55.403 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:55 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:55.403 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:55 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi105", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T02:42:55.404 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:55 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi105", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-06T02:42:55.404 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:55 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:42:55.404 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:55 smithi105 ceph-mon[29228]: Deploying daemon crash.smithi105 on smithi105 2024-04-06T02:42:55.404 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:55 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:56.583 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:56 smithi105 ceph-mon[29228]: Deploying cephadm binary to smithi184 2024-04-06T02:42:56.583 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:56 smithi105 ceph-mon[29228]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:42:56.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:56.791+0000 7ff7fd7fa700 1 -- 172.21.15.105:0/1418692767 <== mgr.14162 v2:172.21.15.105:6800/1949998385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7ff8000611d0 con 0x7ff7e8038410 2024-04-06T02:42:56.777 INFO:teuthology.orchestra.run.smithi105.stdout:Added host 'smithi184' with addr '172.21.15.184' 2024-04-06T02:42:56.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:56.793+0000 7ff8069b2700 1 -- 172.21.15.105:0/1418692767 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7ff7e8038410 msgr2=0x7ff7e803a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:56.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:56.793+0000 7ff8069b2700 1 --2- 172.21.15.105:0/1418692767 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7ff7e8038410 0x7ff7e803a8d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4006fd0 tx=0x7ff7f4006e40 comp rx=0 tx=0).stop 2024-04-06T02:42:56.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:56.793+0000 7ff8069b2700 1 -- 172.21.15.105:0/1418692767 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff8000fef30 msgr2=0x7ff8001959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:56.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:56.793+0000 7ff8069b2700 1 --2- 172.21.15.105:0/1418692767 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff8000fef30 0x7ff8001959c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0005b40 tx=0x7ff7f0004740 comp rx=0 tx=0).stop 2024-04-06T02:42:56.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:56.793+0000 7ff8069b2700 1 -- 172.21.15.105:0/1418692767 shutdown_connections 2024-04-06T02:42:56.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:56.793+0000 7ff8069b2700 1 --2- 172.21.15.105:0/1418692767 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7ff7e8038410 0x7ff7e803a8d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:56.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:56.793+0000 7ff8069b2700 1 --2- 172.21.15.105:0/1418692767 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff8000fef30 0x7ff8001959c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:56.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:56.793+0000 7ff8069b2700 1 -- 172.21.15.105:0/1418692767 >> 172.21.15.105:0/1418692767 conn(0x7ff8000fa4d0 msgr2=0x7ff8000fb190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:56.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:56.794+0000 7ff8069b2700 1 -- 172.21.15.105:0/1418692767 shutdown_connections 2024-04-06T02:42:56.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:56.794+0000 7ff8069b2700 1 -- 172.21.15.105:0/1418692767 wait complete. 2024-04-06T02:42:57.348 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph orch host ls --format=json 2024-04-06T02:42:57.575 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:42:58.008 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:57 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:58.008 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:57 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:58.009 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:57 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:58.009 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:57 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:58.009 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:57 smithi105 ceph-mon[29228]: Deploying daemon node-exporter.smithi105 on smithi105 2024-04-06T02:42:58.009 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:57 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:42:58.009 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:57 smithi105 ceph-mon[29228]: Added host smithi184 2024-04-06T02:42:58.009 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:42:57 smithi105 ceph-mon[29228]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:42:59.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.156+0000 7fd755458700 1 -- 172.21.15.105:0/133131068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd750107f60 msgr2=0x7fd75010a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:59.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.156+0000 7fd755458700 1 --2- 172.21.15.105:0/133131068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd750107f60 0x7fd75010a350 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd748009910 tx=0x7fd748009c20 comp rx=0 tx=0).stop 2024-04-06T02:42:59.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.156+0000 7fd755458700 1 -- 172.21.15.105:0/133131068 shutdown_connections 2024-04-06T02:42:59.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.156+0000 7fd755458700 1 --2- 172.21.15.105:0/133131068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd750107f60 0x7fd75010a350 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:59.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.156+0000 7fd755458700 1 -- 172.21.15.105:0/133131068 >> 172.21.15.105:0/133131068 conn(0x7fd75006daa0 msgr2=0x7fd75006fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:59.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.156+0000 7fd755458700 1 -- 172.21.15.105:0/133131068 shutdown_connections 2024-04-06T02:42:59.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.156+0000 7fd755458700 1 -- 172.21.15.105:0/133131068 wait complete. 2024-04-06T02:42:59.146 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.157+0000 7fd755458700 1 Processor -- start 2024-04-06T02:42:59.146 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.157+0000 7fd755458700 1 -- start start 2024-04-06T02:42:59.146 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.157+0000 7fd755458700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd750107f60 0x7fd75019d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:59.146 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.157+0000 7fd755458700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75019d600 con 0x7fd750107f60 2024-04-06T02:42:59.146 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.157+0000 7fd74effd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd750107f60 0x7fd75019d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:59.146 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.157+0000 7fd74effd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd750107f60 0x7fd75019d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43162/0 (socket says 172.21.15.105:43162) 2024-04-06T02:42:59.146 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.157+0000 7fd74effd700 1 -- 172.21.15.105:0/3539601653 learned_addr learned my addr 172.21.15.105:0/3539601653 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:42:59.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.158+0000 7fd74effd700 1 -- 172.21.15.105:0/3539601653 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7480095f0 con 0x7fd750107f60 2024-04-06T02:42:59.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.158+0000 7fd74effd700 1 --2- 172.21.15.105:0/3539601653 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd750107f60 0x7fd75019d0c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd748009ed0 tx=0x7fd748004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:59.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.158+0000 7fd737fff700 1 -- 172.21.15.105:0/3539601653 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd74801d070 con 0x7fd750107f60 2024-04-06T02:42:59.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.158+0000 7fd737fff700 1 -- 172.21.15.105:0/3539601653 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd74800be10 con 0x7fd750107f60 2024-04-06T02:42:59.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.158+0000 7fd755458700 1 -- 172.21.15.105:0/3539601653 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd75019d800 con 0x7fd750107f60 2024-04-06T02:42:59.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.158+0000 7fd737fff700 1 -- 172.21.15.105:0/3539601653 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd74800f460 con 0x7fd750107f60 2024-04-06T02:42:59.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.158+0000 7fd755458700 1 -- 172.21.15.105:0/3539601653 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd75019dca0 con 0x7fd750107f60 2024-04-06T02:42:59.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.159+0000 7fd737fff700 1 -- 172.21.15.105:0/3539601653 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fd74800f5c0 con 0x7fd750107f60 2024-04-06T02:42:59.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.159+0000 7fd737fff700 1 --2- 172.21.15.105:0/3539601653 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fd738038410 0x7fd73803a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:42:59.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.159+0000 7fd737fff700 1 -- 172.21.15.105:0/3539601653 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd74804d340 con 0x7fd750107f60 2024-04-06T02:42:59.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.159+0000 7fd74e7fc700 1 --2- 172.21.15.105:0/3539601653 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fd738038410 0x7fd73803a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:42:59.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.159+0000 7fd755458700 1 -- 172.21.15.105:0/3539601653 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7500623c0 con 0x7fd750107f60 2024-04-06T02:42:59.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.160+0000 7fd74e7fc700 1 --2- 172.21.15.105:0/3539601653 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fd738038410 0x7fd73803a8d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd74400ad30 tx=0x7fd7440093f0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:42:59.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.164+0000 7fd737fff700 1 -- 172.21.15.105:0/3539601653 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd748027070 con 0x7fd750107f60 2024-04-06T02:42:59.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.366+0000 7fd755458700 1 -- 172.21.15.105:0/3539601653 --> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd75006f6f0 con 0x7fd738038410 2024-04-06T02:42:59.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.367+0000 7fd737fff700 1 -- 172.21.15.105:0/3539601653 <== mgr.14162 v2:172.21.15.105:6800/1949998385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7fd75006f6f0 con 0x7fd738038410 2024-04-06T02:42:59.351 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:42:59.351 INFO:teuthology.orchestra.run.smithi105.stdout:[{"addr": "172.21.15.105", "hostname": "smithi105", "labels": [], "status": ""}, {"addr": "172.21.15.184", "hostname": "smithi184", "labels": [], "status": ""}] 2024-04-06T02:42:59.353 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.369+0000 7fd755458700 1 -- 172.21.15.105:0/3539601653 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fd738038410 msgr2=0x7fd73803a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:59.353 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.369+0000 7fd755458700 1 --2- 172.21.15.105:0/3539601653 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fd738038410 0x7fd73803a8d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd74400ad30 tx=0x7fd7440093f0 comp rx=0 tx=0).stop 2024-04-06T02:42:59.353 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.369+0000 7fd755458700 1 -- 172.21.15.105:0/3539601653 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd750107f60 msgr2=0x7fd75019d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:42:59.353 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.369+0000 7fd755458700 1 --2- 172.21.15.105:0/3539601653 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd750107f60 0x7fd75019d0c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd748009ed0 tx=0x7fd748004690 comp rx=0 tx=0).stop 2024-04-06T02:42:59.354 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.369+0000 7fd755458700 1 -- 172.21.15.105:0/3539601653 shutdown_connections 2024-04-06T02:42:59.354 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.369+0000 7fd755458700 1 --2- 172.21.15.105:0/3539601653 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fd738038410 0x7fd73803a8d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:59.354 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.369+0000 7fd755458700 1 --2- 172.21.15.105:0/3539601653 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd750107f60 0x7fd75019d0c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:42:59.354 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.369+0000 7fd755458700 1 -- 172.21.15.105:0/3539601653 >> 172.21.15.105:0/3539601653 conn(0x7fd75006daa0 msgr2=0x7fd75006efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:42:59.354 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.369+0000 7fd755458700 1 -- 172.21.15.105:0/3539601653 shutdown_connections 2024-04-06T02:42:59.354 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:42:59.369+0000 7fd755458700 1 -- 172.21.15.105:0/3539601653 wait complete. 2024-04-06T02:43:00.132 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-06T02:43:00.132 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph osd crush tunables default 2024-04-06T02:43:00.335 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:43:00.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:00 smithi105 ceph-mon[29228]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:43:00.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:00 smithi105 ceph-mon[29228]: from='client.14191 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T02:43:00.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:00 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:02.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:02 smithi105 ceph-mon[29228]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:43:02.916 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.931+0000 7f5716847700 1 -- 172.21.15.105:0/3812628635 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57100feb60 msgr2=0x7f57100fef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:02.916 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.931+0000 7f5716847700 1 --2- 172.21.15.105:0/3812628635 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57100feb60 0x7f57100fef80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5700009910 tx=0x7f5700009c20 comp rx=0 tx=0).stop 2024-04-06T02:43:02.916 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.931+0000 7f5716847700 1 -- 172.21.15.105:0/3812628635 shutdown_connections 2024-04-06T02:43:02.916 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.931+0000 7f5716847700 1 --2- 172.21.15.105:0/3812628635 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57100feb60 0x7f57100fef80 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:02.916 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.931+0000 7f5716847700 1 -- 172.21.15.105:0/3812628635 >> 172.21.15.105:0/3812628635 conn(0x7f57100fa4a0 msgr2=0x7f57100fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:02.916 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.931+0000 7f5716847700 1 -- 172.21.15.105:0/3812628635 shutdown_connections 2024-04-06T02:43:02.916 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.931+0000 7f5716847700 1 -- 172.21.15.105:0/3812628635 wait complete. 2024-04-06T02:43:02.916 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.931+0000 7f5716847700 1 Processor -- start 2024-04-06T02:43:02.917 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.932+0000 7f5716847700 1 -- start start 2024-04-06T02:43:02.918 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.932+0000 7f5716847700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57100feb60 0x7f57101959b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:02.918 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.932+0000 7f5716847700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5710195ef0 con 0x7f57100feb60 2024-04-06T02:43:02.918 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.932+0000 7f570ffff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57100feb60 0x7f57101959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:02.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.932+0000 7f570ffff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57100feb60 0x7f57101959b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:52014/0 (socket says 172.21.15.105:52014) 2024-04-06T02:43:02.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.932+0000 7f570ffff700 1 -- 172.21.15.105:0/4063830991 learned_addr learned my addr 172.21.15.105:0/4063830991 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:43:02.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.932+0000 7f570ffff700 1 -- 172.21.15.105:0/4063830991 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57000095f0 con 0x7f57100feb60 2024-04-06T02:43:02.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.932+0000 7f570ffff700 1 --2- 172.21.15.105:0/4063830991 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57100feb60 0x7f57101959b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5700005950 tx=0x7f5700009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:02.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.933+0000 7f570d7fa700 1 -- 172.21.15.105:0/4063830991 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f570001d070 con 0x7f57100feb60 2024-04-06T02:43:02.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.933+0000 7f5716847700 1 -- 172.21.15.105:0/4063830991 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57101960f0 con 0x7f57100feb60 2024-04-06T02:43:02.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.933+0000 7f570d7fa700 1 -- 172.21.15.105:0/4063830991 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f570000be10 con 0x7f57100feb60 2024-04-06T02:43:02.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.933+0000 7f570d7fa700 1 -- 172.21.15.105:0/4063830991 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f570000f460 con 0x7f57100feb60 2024-04-06T02:43:02.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.933+0000 7f5716847700 1 -- 172.21.15.105:0/4063830991 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5710196590 con 0x7f57100feb60 2024-04-06T02:43:02.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.933+0000 7f570d7fa700 1 -- 172.21.15.105:0/4063830991 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f570000f690 con 0x7f57100feb60 2024-04-06T02:43:02.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.934+0000 7f5716847700 1 -- 172.21.15.105:0/4063830991 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f571018f4d0 con 0x7f57100feb60 2024-04-06T02:43:02.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.934+0000 7f570d7fa700 1 --2- 172.21.15.105:0/4063830991 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f56f80383c0 0x7f56f803a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:02.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.934+0000 7f570d7fa700 1 -- 172.21.15.105:0/4063830991 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f570004c5a0 con 0x7f57100feb60 2024-04-06T02:43:02.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.934+0000 7f570f7fe700 1 --2- 172.21.15.105:0/4063830991 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f56f80383c0 0x7f56f803a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:02.920 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.934+0000 7f570f7fe700 1 --2- 172.21.15.105:0/4063830991 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f56f80383c0 0x7f56f803a880 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5704006fd0 tx=0x7f5704006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:02.923 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:02.939+0000 7f570d7fa700 1 -- 172.21.15.105:0/4063830991 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5700020730 con 0x7f57100feb60 2024-04-06T02:43:03.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:03.139+0000 7f5716847700 1 -- 172.21.15.105:0/4063830991 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f57100ff690 con 0x7f57100feb60 2024-04-06T02:43:04.096 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:04.112+0000 7f570d7fa700 1 -- 172.21.15.105:0/4063830991 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f57100ff690 con 0x7f57100feb60 2024-04-06T02:43:04.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:04.114+0000 7f5716847700 1 -- 172.21.15.105:0/4063830991 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f56f80383c0 msgr2=0x7f56f803a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:04.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:04.114+0000 7f5716847700 1 --2- 172.21.15.105:0/4063830991 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f56f80383c0 0x7f56f803a880 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5704006fd0 tx=0x7f5704006e40 comp rx=0 tx=0).stop 2024-04-06T02:43:04.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:04.114+0000 7f5716847700 1 -- 172.21.15.105:0/4063830991 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57100feb60 msgr2=0x7f57101959b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:04.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:04.114+0000 7f5716847700 1 --2- 172.21.15.105:0/4063830991 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57100feb60 0x7f57101959b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5700005950 tx=0x7f5700009e60 comp rx=0 tx=0).stop 2024-04-06T02:43:04.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:04.114+0000 7f5716847700 1 -- 172.21.15.105:0/4063830991 shutdown_connections 2024-04-06T02:43:04.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:04.114+0000 7f5716847700 1 --2- 172.21.15.105:0/4063830991 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f56f80383c0 0x7f56f803a880 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:04.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:04.114+0000 7f5716847700 1 --2- 172.21.15.105:0/4063830991 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57100feb60 0x7f57101959b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:04.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:04.114+0000 7f5716847700 1 -- 172.21.15.105:0/4063830991 >> 172.21.15.105:0/4063830991 conn(0x7f57100fa4a0 msgr2=0x7f57100fb180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:04.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:04.114+0000 7f5716847700 1 -- 172.21.15.105:0/4063830991 shutdown_connections 2024-04-06T02:43:04.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:43:04.114+0000 7f5716847700 1 -- 172.21.15.105:0/4063830991 wait complete. 2024-04-06T02:43:04.100 INFO:teuthology.orchestra.run.smithi105.stderr:adjusted tunables profile to default 2024-04-06T02:43:04.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:04 smithi105 ceph-mon[29228]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:43:04.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:04 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:04.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:04 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:04.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:04 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:04.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:04 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:04.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:04 smithi105 ceph-mon[29228]: Deploying daemon alertmanager.smithi105 on smithi105 2024-04-06T02:43:04.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:04 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/4063830991' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-06T02:43:04.996 INFO:tasks.cephadm:Adding mon.smithi105 on smithi105 2024-04-06T02:43:04.996 INFO:tasks.cephadm:Adding mon.smithi184 on smithi184 2024-04-06T02:43:04.996 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph orch apply mon '2;smithi105:172.21.15.105=smithi105;smithi184:172.21.15.184=smithi184' 2024-04-06T02:43:05.199 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:05.244 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:05.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:05 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/4063830991' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-06T02:43:05.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:05 smithi105 ceph-mon[29228]: osdmap e4: 0 total, 0 up, 0 in 2024-04-06T02:43:05.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:05 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:06.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:06 smithi105 ceph-mon[29228]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:43:06.859 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.873+0000 7f4d35edf700 1 -- 172.21.15.184:0/1892586241 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d30101320 msgr2=0x7f4d30103710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:06.859 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.873+0000 7f4d35edf700 1 --2- 172.21.15.184:0/1892586241 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d30101320 0x7f4d30103710 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4d20009910 tx=0x7f4d20009c20 comp rx=0 tx=0).stop 2024-04-06T02:43:06.860 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.874+0000 7f4d35edf700 1 -- 172.21.15.184:0/1892586241 shutdown_connections 2024-04-06T02:43:06.860 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.874+0000 7f4d35edf700 1 --2- 172.21.15.184:0/1892586241 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d30101320 0x7f4d30103710 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:06.860 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.874+0000 7f4d35edf700 1 -- 172.21.15.184:0/1892586241 >> 172.21.15.184:0/1892586241 conn(0x7f4d300fad30 msgr2=0x7f4d300fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:06.860 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.874+0000 7f4d35edf700 1 -- 172.21.15.184:0/1892586241 shutdown_connections 2024-04-06T02:43:06.860 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.874+0000 7f4d35edf700 1 -- 172.21.15.184:0/1892586241 wait complete. 2024-04-06T02:43:06.860 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.874+0000 7f4d35edf700 1 Processor -- start 2024-04-06T02:43:06.860 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.874+0000 7f4d35edf700 1 -- start start 2024-04-06T02:43:06.860 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.874+0000 7f4d35edf700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d30101320 0x7f4d30197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:06.860 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.874+0000 7f4d35edf700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d301980a0 con 0x7f4d30101320 2024-04-06T02:43:06.860 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.875+0000 7f4d2f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d30101320 0x7f4d30197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:06.860 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.875+0000 7f4d2f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d30101320 0x7f4d30197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.184:50856/0 (socket says 172.21.15.184:50856) 2024-04-06T02:43:06.860 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.875+0000 7f4d2f7fe700 1 -- 172.21.15.184:0/2412600282 learned_addr learned my addr 172.21.15.184:0/2412600282 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-04-06T02:43:06.860 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.875+0000 7f4d2f7fe700 1 -- 172.21.15.184:0/2412600282 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d200095f0 con 0x7f4d30101320 2024-04-06T02:43:06.862 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.875+0000 7f4d2f7fe700 1 --2- 172.21.15.184:0/2412600282 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d30101320 0x7f4d30197b60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4d20005b40 tx=0x7f4d20004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:06.862 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.875+0000 7f4d2cff9700 1 -- 172.21.15.184:0/2412600282 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d2001d070 con 0x7f4d30101320 2024-04-06T02:43:06.862 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.875+0000 7f4d35edf700 1 -- 172.21.15.184:0/2412600282 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d301982a0 con 0x7f4d30101320 2024-04-06T02:43:06.862 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.875+0000 7f4d2cff9700 1 -- 172.21.15.184:0/2412600282 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4d2000be10 con 0x7f4d30101320 2024-04-06T02:43:06.862 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.875+0000 7f4d2cff9700 1 -- 172.21.15.184:0/2412600282 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d2000f460 con 0x7f4d30101320 2024-04-06T02:43:06.862 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.875+0000 7f4d35edf700 1 -- 172.21.15.184:0/2412600282 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d30198740 con 0x7f4d30101320 2024-04-06T02:43:06.862 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.876+0000 7f4d2cff9700 1 -- 172.21.15.184:0/2412600282 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f4d2000f6d0 con 0x7f4d30101320 2024-04-06T02:43:06.862 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.876+0000 7f4d2cff9700 1 --2- 172.21.15.184:0/2412600282 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f4d180383c0 0x7f4d1803a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:06.862 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.876+0000 7f4d2cff9700 1 -- 172.21.15.184:0/2412600282 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4d2004c4e0 con 0x7f4d30101320 2024-04-06T02:43:06.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.876+0000 7f4d35edf700 1 -- 172.21.15.184:0/2412600282 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d301918b0 con 0x7f4d30101320 2024-04-06T02:43:06.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.877+0000 7f4d2effd700 1 --2- 172.21.15.184:0/2412600282 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f4d180383c0 0x7f4d1803a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:06.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.877+0000 7f4d2effd700 1 --2- 172.21.15.184:0/2412600282 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f4d180383c0 0x7f4d1803a880 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4d24006fd0 tx=0x7f4d24006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:06.866 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:06.882+0000 7f4d2cff9700 1 -- 172.21.15.184:0/2412600282 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4d20027070 con 0x7f4d30101320 2024-04-06T02:43:07.063 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:07.078+0000 7f4d35edf700 1 -- 172.21.15.184:0/2412600282 --> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi105:172.21.15.105=smithi105;smithi184:172.21.15.184=smithi184", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d30061340 con 0x7f4d180383c0 2024-04-06T02:43:07.069 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:07.085+0000 7f4d2cff9700 1 -- 172.21.15.184:0/2412600282 <== mgr.14162 v2:172.21.15.105:6800/1949998385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f4d30061340 con 0x7f4d180383c0 2024-04-06T02:43:07.070 INFO:teuthology.orchestra.run.smithi184.stdout:Scheduled mon update... 2024-04-06T02:43:07.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:07.088+0000 7f4d35edf700 1 -- 172.21.15.184:0/2412600282 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f4d180383c0 msgr2=0x7f4d1803a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:07.073 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:07.088+0000 7f4d35edf700 1 --2- 172.21.15.184:0/2412600282 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f4d180383c0 0x7f4d1803a880 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4d24006fd0 tx=0x7f4d24006e40 comp rx=0 tx=0).stop 2024-04-06T02:43:07.073 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:07.088+0000 7f4d35edf700 1 -- 172.21.15.184:0/2412600282 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d30101320 msgr2=0x7f4d30197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:07.073 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:07.088+0000 7f4d35edf700 1 --2- 172.21.15.184:0/2412600282 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d30101320 0x7f4d30197b60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4d20005b40 tx=0x7f4d20004740 comp rx=0 tx=0).stop 2024-04-06T02:43:07.073 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:07.088+0000 7f4d35edf700 1 -- 172.21.15.184:0/2412600282 shutdown_connections 2024-04-06T02:43:07.073 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:07.088+0000 7f4d35edf700 1 --2- 172.21.15.184:0/2412600282 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f4d180383c0 0x7f4d1803a880 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:07.073 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:07.088+0000 7f4d35edf700 1 --2- 172.21.15.184:0/2412600282 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d30101320 0x7f4d30197b60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:07.073 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:07.088+0000 7f4d35edf700 1 -- 172.21.15.184:0/2412600282 >> 172.21.15.184:0/2412600282 conn(0x7f4d300fad30 msgr2=0x7f4d300fb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:07.073 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:07.088+0000 7f4d35edf700 1 -- 172.21.15.184:0/2412600282 shutdown_connections 2024-04-06T02:43:07.073 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:07.088+0000 7f4d35edf700 1 -- 172.21.15.184:0/2412600282 wait complete. 2024-04-06T02:43:07.643 DEBUG:teuthology.orchestra.run.smithi184:mon.smithi184> sudo journalctl -f -n 0 -u ceph-130df578-f3bf-11ee-b649-cb9ed24678a4@mon.smithi184.service 2024-04-06T02:43:07.646 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T02:43:07.646 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T02:43:07.873 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:07.918 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:08.402 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:08 smithi105 ceph-mon[29228]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:43:08.402 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:08 smithi105 ceph-mon[29228]: from='client.14195 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi105:172.21.15.105=smithi105;smithi184:172.21.15.184=smithi184", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:43:08.402 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:08 smithi105 ceph-mon[29228]: Saving service mon spec with placement smithi105:172.21.15.105=smithi105;smithi184:172.21.15.184=smithi184;count:2 2024-04-06T02:43:08.402 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:08 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:09.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.301+0000 7f937115a700 1 -- 172.21.15.184:0/169322819 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f936c1008b0 msgr2=0x7f936c100cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:09.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.301+0000 7f937115a700 1 --2- 172.21.15.184:0/169322819 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f936c1008b0 0x7f936c100cd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f935c009910 tx=0x7f935c009c20 comp rx=0 tx=0).stop 2024-04-06T02:43:09.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.301+0000 7f937115a700 1 -- 172.21.15.184:0/169322819 shutdown_connections 2024-04-06T02:43:09.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.301+0000 7f937115a700 1 --2- 172.21.15.184:0/169322819 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f936c1008b0 0x7f936c100cd0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:09.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.301+0000 7f937115a700 1 -- 172.21.15.184:0/169322819 >> 172.21.15.184:0/169322819 conn(0x7f936c0fbe30 msgr2=0x7f936c0fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:09.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.301+0000 7f937115a700 1 -- 172.21.15.184:0/169322819 shutdown_connections 2024-04-06T02:43:09.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.301+0000 7f937115a700 1 -- 172.21.15.184:0/169322819 wait complete. 2024-04-06T02:43:09.287 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.302+0000 7f937115a700 1 Processor -- start 2024-04-06T02:43:09.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.302+0000 7f937115a700 1 -- start start 2024-04-06T02:43:09.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.302+0000 7f937115a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f936c1008b0 0x7f936c1959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:09.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.302+0000 7f937115a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936c195f00 con 0x7f936c1008b0 2024-04-06T02:43:09.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.302+0000 7f936ad9d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f936c1008b0 0x7f936c1959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:09.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.302+0000 7f936ad9d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f936c1008b0 0x7f936c1959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.184:50886/0 (socket says 172.21.15.184:50886) 2024-04-06T02:43:09.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.302+0000 7f936ad9d700 1 -- 172.21.15.184:0/1490902660 learned_addr learned my addr 172.21.15.184:0/1490902660 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-04-06T02:43:09.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.303+0000 7f936ad9d700 1 -- 172.21.15.184:0/1490902660 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935c0095f0 con 0x7f936c1008b0 2024-04-06T02:43:09.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.303+0000 7f936ad9d700 1 --2- 172.21.15.184:0/1490902660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f936c1008b0 0x7f936c1959c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f935c005b40 tx=0x7f935c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:09.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.303+0000 7f9353fff700 1 -- 172.21.15.184:0/1490902660 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f935c01d070 con 0x7f936c1008b0 2024-04-06T02:43:09.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.303+0000 7f937115a700 1 -- 172.21.15.184:0/1490902660 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f936c196100 con 0x7f936c1008b0 2024-04-06T02:43:09.289 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.303+0000 7f9353fff700 1 -- 172.21.15.184:0/1490902660 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f935c00be10 con 0x7f936c1008b0 2024-04-06T02:43:09.289 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.303+0000 7f9353fff700 1 -- 172.21.15.184:0/1490902660 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f935c00f460 con 0x7f936c1008b0 2024-04-06T02:43:09.289 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.303+0000 7f937115a700 1 -- 172.21.15.184:0/1490902660 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f936c1965a0 con 0x7f936c1008b0 2024-04-06T02:43:09.290 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.304+0000 7f937115a700 1 -- 172.21.15.184:0/1490902660 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f936c18f4d0 con 0x7f936c1008b0 2024-04-06T02:43:09.290 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.304+0000 7f9353fff700 1 -- 172.21.15.184:0/1490902660 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f935c00b960 con 0x7f936c1008b0 2024-04-06T02:43:09.290 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.304+0000 7f9353fff700 1 --2- 172.21.15.184:0/1490902660 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f9354038410 0x7f935403a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:09.290 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.304+0000 7f9353fff700 1 -- 172.21.15.184:0/1490902660 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f935c04c420 con 0x7f936c1008b0 2024-04-06T02:43:09.290 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.305+0000 7f936a59c700 1 --2- 172.21.15.184:0/1490902660 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f9354038410 0x7f935403a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:09.290 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.305+0000 7f936a59c700 1 --2- 172.21.15.184:0/1490902660 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f9354038410 0x7f935403a8d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9360006fd0 tx=0x7f9360006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:09.294 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.309+0000 7f9353fff700 1 -- 172.21.15.184:0/1490902660 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f935c02bb60 con 0x7f936c1008b0 2024-04-06T02:43:09.559 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.574+0000 7f937115a700 1 -- 172.21.15.184:0/1490902660 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f936c02cc70 con 0x7f936c1008b0 2024-04-06T02:43:09.560 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.575+0000 7f9353fff700 1 -- 172.21.15.184:0/1490902660 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f935c027030 con 0x7f936c1008b0 2024-04-06T02:43:09.560 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:43:09.560 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"130df578-f3bf-11ee-b649-cb9ed24678a4","modified":"2024-04-06T02:41:29.716000Z","created":"2024-04-06T02:41:29.716000Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T02:43:09.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.577+0000 7f937115a700 1 -- 172.21.15.184:0/1490902660 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f9354038410 msgr2=0x7f935403a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:09.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.577+0000 7f937115a700 1 --2- 172.21.15.184:0/1490902660 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f9354038410 0x7f935403a8d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9360006fd0 tx=0x7f9360006e40 comp rx=0 tx=0).stop 2024-04-06T02:43:09.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.578+0000 7f937115a700 1 -- 172.21.15.184:0/1490902660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f936c1008b0 msgr2=0x7f936c1959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:09.563 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.578+0000 7f937115a700 1 --2- 172.21.15.184:0/1490902660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f936c1008b0 0x7f936c1959c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f935c005b40 tx=0x7f935c004740 comp rx=0 tx=0).stop 2024-04-06T02:43:09.563 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.578+0000 7f937115a700 1 -- 172.21.15.184:0/1490902660 shutdown_connections 2024-04-06T02:43:09.563 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.578+0000 7f937115a700 1 --2- 172.21.15.184:0/1490902660 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f9354038410 0x7f935403a8d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:09.563 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.578+0000 7f937115a700 1 --2- 172.21.15.184:0/1490902660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f936c1008b0 0x7f936c1959c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:09.563 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.578+0000 7f937115a700 1 -- 172.21.15.184:0/1490902660 >> 172.21.15.184:0/1490902660 conn(0x7f936c0fbe30 msgr2=0x7f936c0fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:09.563 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.578+0000 7f937115a700 1 -- 172.21.15.184:0/1490902660 shutdown_connections 2024-04-06T02:43:09.563 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:09.578+0000 7f937115a700 1 -- 172.21.15.184:0/1490902660 wait complete. 2024-04-06T02:43:09.563 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-04-06T02:43:10.386 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:10 smithi105 ceph-mon[29228]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:43:10.386 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:10 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/1490902660' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T02:43:11.151 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T02:43:11.151 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T02:43:11.351 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:11.396 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:12.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:12 smithi105 ceph-mon[29228]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:43:12.809 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.823+0000 7f315c0fc700 1 -- 172.21.15.184:0/1022604098 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31540ff790 msgr2=0x7f31540ffbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:12.810 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.823+0000 7f315c0fc700 1 --2- 172.21.15.184:0/1022604098 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31540ff790 0x7f31540ffbb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3144009910 tx=0x7f3144009c20 comp rx=0 tx=0).stop 2024-04-06T02:43:12.810 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.823+0000 7f315c0fc700 1 -- 172.21.15.184:0/1022604098 shutdown_connections 2024-04-06T02:43:12.810 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.823+0000 7f315c0fc700 1 --2- 172.21.15.184:0/1022604098 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31540ff790 0x7f31540ffbb0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:12.810 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.823+0000 7f315c0fc700 1 -- 172.21.15.184:0/1022604098 >> 172.21.15.184:0/1022604098 conn(0x7f31540fad30 msgr2=0x7f31540fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:12.810 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.823+0000 7f315c0fc700 1 -- 172.21.15.184:0/1022604098 shutdown_connections 2024-04-06T02:43:12.810 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.823+0000 7f315c0fc700 1 -- 172.21.15.184:0/1022604098 wait complete. 2024-04-06T02:43:12.810 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.823+0000 7f315c0fc700 1 Processor -- start 2024-04-06T02:43:12.810 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.823+0000 7f315c0fc700 1 -- start start 2024-04-06T02:43:12.810 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.824+0000 7f315c0fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31540ff790 0x7f3154069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:12.810 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.824+0000 7f315c0fc700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31541986c0 con 0x7f31540ff790 2024-04-06T02:43:12.810 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.824+0000 7f3159e98700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31540ff790 0x7f3154069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:12.810 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.824+0000 7f3159e98700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31540ff790 0x7f3154069410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.184:47130/0 (socket says 172.21.15.184:47130) 2024-04-06T02:43:12.811 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.824+0000 7f3159e98700 1 -- 172.21.15.184:0/1364925136 learned_addr learned my addr 172.21.15.184:0/1364925136 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-04-06T02:43:12.811 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.824+0000 7f3159e98700 1 -- 172.21.15.184:0/1364925136 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31440095f0 con 0x7f31540ff790 2024-04-06T02:43:12.811 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.824+0000 7f3159e98700 1 --2- 172.21.15.184:0/1364925136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31540ff790 0x7f3154069410 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3144005b40 tx=0x7f3144004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:12.811 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.824+0000 7f314affd700 1 -- 172.21.15.184:0/1364925136 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f314401d070 con 0x7f31540ff790 2024-04-06T02:43:12.811 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.825+0000 7f314affd700 1 -- 172.21.15.184:0/1364925136 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f314400be10 con 0x7f31540ff790 2024-04-06T02:43:12.811 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.825+0000 7f315c0fc700 1 -- 172.21.15.184:0/1364925136 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3154198800 con 0x7f31540ff790 2024-04-06T02:43:12.811 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.825+0000 7f314affd700 1 -- 172.21.15.184:0/1364925136 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f314400f460 con 0x7f31540ff790 2024-04-06T02:43:12.811 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.825+0000 7f315c0fc700 1 -- 172.21.15.184:0/1364925136 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3154198ca0 con 0x7f31540ff790 2024-04-06T02:43:12.812 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.825+0000 7f314affd700 1 -- 172.21.15.184:0/1364925136 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f314400b960 con 0x7f31540ff790 2024-04-06T02:43:12.812 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.825+0000 7f315c0fc700 1 -- 172.21.15.184:0/1364925136 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f315418e3b0 con 0x7f31540ff790 2024-04-06T02:43:12.812 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.826+0000 7f314affd700 1 --2- 172.21.15.184:0/1364925136 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f31400383c0 0x7f314003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:12.812 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.826+0000 7f314affd700 1 -- 172.21.15.184:0/1364925136 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f314404c440 con 0x7f31540ff790 2024-04-06T02:43:12.812 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.826+0000 7f3159697700 1 --2- 172.21.15.184:0/1364925136 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f31400383c0 0x7f314003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:12.812 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.826+0000 7f3159697700 1 --2- 172.21.15.184:0/1364925136 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f31400383c0 0x7f314003a880 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3150006fd0 tx=0x7f3150006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:12.815 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:12.831+0000 7f314affd700 1 -- 172.21.15.184:0/1364925136 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f314402a360 con 0x7f31540ff790 2024-04-06T02:43:13.081 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:13.096+0000 7f315c0fc700 1 -- 172.21.15.184:0/1364925136 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f315402cc70 con 0x7f31540ff790 2024-04-06T02:43:13.081 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:13.096+0000 7f314affd700 1 -- 172.21.15.184:0/1364925136 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f3144027030 con 0x7f31540ff790 2024-04-06T02:43:13.081 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:43:13.081 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"130df578-f3bf-11ee-b649-cb9ed24678a4","modified":"2024-04-06T02:41:29.716000Z","created":"2024-04-06T02:41:29.716000Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T02:43:13.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:13.098+0000 7f315c0fc700 1 -- 172.21.15.184:0/1364925136 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f31400383c0 msgr2=0x7f314003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:13.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:13.098+0000 7f315c0fc700 1 --2- 172.21.15.184:0/1364925136 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f31400383c0 0x7f314003a880 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3150006fd0 tx=0x7f3150006e40 comp rx=0 tx=0).stop 2024-04-06T02:43:13.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:13.098+0000 7f315c0fc700 1 -- 172.21.15.184:0/1364925136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31540ff790 msgr2=0x7f3154069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:13.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:13.098+0000 7f315c0fc700 1 --2- 172.21.15.184:0/1364925136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31540ff790 0x7f3154069410 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3144005b40 tx=0x7f3144004740 comp rx=0 tx=0).stop 2024-04-06T02:43:13.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:13.098+0000 7f315c0fc700 1 -- 172.21.15.184:0/1364925136 shutdown_connections 2024-04-06T02:43:13.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:13.098+0000 7f315c0fc700 1 --2- 172.21.15.184:0/1364925136 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f31400383c0 0x7f314003a880 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:13.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:13.098+0000 7f315c0fc700 1 --2- 172.21.15.184:0/1364925136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31540ff790 0x7f3154069410 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:13.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:13.098+0000 7f315c0fc700 1 -- 172.21.15.184:0/1364925136 >> 172.21.15.184:0/1364925136 conn(0x7f31540fad30 msgr2=0x7f31540fb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:13.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:13.098+0000 7f315c0fc700 1 -- 172.21.15.184:0/1364925136 shutdown_connections 2024-04-06T02:43:13.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:13.098+0000 7f315c0fc700 1 -- 172.21.15.184:0/1364925136 wait complete. 2024-04-06T02:43:13.084 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-04-06T02:43:13.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:13 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:13.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:13 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:13.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:13 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:13.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:13 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:13.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:13 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:13.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:13 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:13.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:13 smithi105 ceph-mon[29228]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-06T02:43:13.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:13 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:13.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:13 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:13.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:13 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T02:43:13.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:13 smithi105 ceph-mon[29228]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T02:43:13.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:13 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:13.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:13 smithi105 ceph-mon[29228]: Deploying daemon grafana.smithi105 on smithi105 2024-04-06T02:43:13.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:13 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/1364925136' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T02:43:14.516 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T02:43:14.517 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T02:43:14.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:14 smithi105 ceph-mon[29228]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:43:14.719 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:14.765 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:16.309 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.323+0000 7f0018613700 1 -- 172.21.15.184:0/2634994073 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00100ff790 msgr2=0x7f00100ffbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:16.309 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.323+0000 7f0018613700 1 --2- 172.21.15.184:0/2634994073 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00100ff790 0x7f00100ffbb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f000c009910 tx=0x7f000c009c20 comp rx=0 tx=0).stop 2024-04-06T02:43:16.310 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.323+0000 7f0018613700 1 -- 172.21.15.184:0/2634994073 shutdown_connections 2024-04-06T02:43:16.310 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.323+0000 7f0018613700 1 --2- 172.21.15.184:0/2634994073 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00100ff790 0x7f00100ffbb0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:16.310 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.323+0000 7f0018613700 1 -- 172.21.15.184:0/2634994073 >> 172.21.15.184:0/2634994073 conn(0x7f00100fad30 msgr2=0x7f00100fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:16.310 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.323+0000 7f0018613700 1 -- 172.21.15.184:0/2634994073 shutdown_connections 2024-04-06T02:43:16.310 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.323+0000 7f0018613700 1 -- 172.21.15.184:0/2634994073 wait complete. 2024-04-06T02:43:16.310 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.323+0000 7f0018613700 1 Processor -- start 2024-04-06T02:43:16.310 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.323+0000 7f0018613700 1 -- start start 2024-04-06T02:43:16.310 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.324+0000 7f0018613700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00100ff790 0x7f00100693f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:16.310 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.324+0000 7f0018613700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00101964d0 con 0x7f00100ff790 2024-04-06T02:43:16.310 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.324+0000 7f00163af700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00100ff790 0x7f00100693f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:16.310 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.324+0000 7f00163af700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00100ff790 0x7f00100693f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.184:47152/0 (socket says 172.21.15.184:47152) 2024-04-06T02:43:16.311 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.324+0000 7f00163af700 1 -- 172.21.15.184:0/2002943819 learned_addr learned my addr 172.21.15.184:0/2002943819 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-04-06T02:43:16.311 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.324+0000 7f00163af700 1 -- 172.21.15.184:0/2002943819 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f000c0095f0 con 0x7f00100ff790 2024-04-06T02:43:16.311 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.324+0000 7f00163af700 1 --2- 172.21.15.184:0/2002943819 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00100ff790 0x7f00100693f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f000c005b40 tx=0x7f000c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:16.311 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.324+0000 7f00077fe700 1 -- 172.21.15.184:0/2002943819 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f000c01d070 con 0x7f00100ff790 2024-04-06T02:43:16.311 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.324+0000 7f00077fe700 1 -- 172.21.15.184:0/2002943819 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f000c00be10 con 0x7f00100ff790 2024-04-06T02:43:16.311 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.324+0000 7f0018613700 1 -- 172.21.15.184:0/2002943819 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f001019aae0 con 0x7f00100ff790 2024-04-06T02:43:16.311 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.324+0000 7f00077fe700 1 -- 172.21.15.184:0/2002943819 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f000c00f460 con 0x7f00100ff790 2024-04-06T02:43:16.311 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.324+0000 7f0018613700 1 -- 172.21.15.184:0/2002943819 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f001019ae40 con 0x7f00100ff790 2024-04-06T02:43:16.312 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.325+0000 7f00077fe700 1 -- 172.21.15.184:0/2002943819 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f000c00f6d0 con 0x7f00100ff790 2024-04-06T02:43:16.312 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.325+0000 7f0018613700 1 -- 172.21.15.184:0/2002943819 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f001018e3b0 con 0x7f00100ff790 2024-04-06T02:43:16.312 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.325+0000 7f00077fe700 1 --2- 172.21.15.184:0/2002943819 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7efffc0383c0 0x7efffc03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:16.312 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.325+0000 7f00077fe700 1 -- 172.21.15.184:0/2002943819 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f000c04c540 con 0x7f00100ff790 2024-04-06T02:43:16.312 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.326+0000 7f0015bae700 1 --2- 172.21.15.184:0/2002943819 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7efffc0383c0 0x7efffc03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:16.312 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.326+0000 7f0015bae700 1 --2- 172.21.15.184:0/2002943819 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7efffc0383c0 0x7efffc03a880 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0000006fd0 tx=0x7f0000006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:16.317 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.332+0000 7f00077fe700 1 -- 172.21.15.184:0/2002943819 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f000c02a360 con 0x7f00100ff790 2024-04-06T02:43:16.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:16 smithi105 ceph-mon[29228]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:43:16.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:16 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:16.582 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.598+0000 7f0018613700 1 -- 172.21.15.184:0/2002943819 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f001002cc70 con 0x7f00100ff790 2024-04-06T02:43:16.583 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.598+0000 7f00077fe700 1 -- 172.21.15.184:0/2002943819 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f000c027030 con 0x7f00100ff790 2024-04-06T02:43:16.583 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:43:16.583 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"130df578-f3bf-11ee-b649-cb9ed24678a4","modified":"2024-04-06T02:41:29.716000Z","created":"2024-04-06T02:41:29.716000Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T02:43:16.585 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.600+0000 7f0018613700 1 -- 172.21.15.184:0/2002943819 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7efffc0383c0 msgr2=0x7efffc03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:16.586 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.600+0000 7f0018613700 1 --2- 172.21.15.184:0/2002943819 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7efffc0383c0 0x7efffc03a880 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0000006fd0 tx=0x7f0000006e40 comp rx=0 tx=0).stop 2024-04-06T02:43:16.586 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.600+0000 7f0018613700 1 -- 172.21.15.184:0/2002943819 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00100ff790 msgr2=0x7f00100693f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:16.586 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.600+0000 7f0018613700 1 --2- 172.21.15.184:0/2002943819 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00100ff790 0x7f00100693f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f000c005b40 tx=0x7f000c004740 comp rx=0 tx=0).stop 2024-04-06T02:43:16.586 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.600+0000 7f0018613700 1 -- 172.21.15.184:0/2002943819 shutdown_connections 2024-04-06T02:43:16.586 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.600+0000 7f0018613700 1 --2- 172.21.15.184:0/2002943819 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7efffc0383c0 0x7efffc03a880 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:16.586 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.600+0000 7f0018613700 1 --2- 172.21.15.184:0/2002943819 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00100ff790 0x7f00100693f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:16.586 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.600+0000 7f0018613700 1 -- 172.21.15.184:0/2002943819 >> 172.21.15.184:0/2002943819 conn(0x7f00100fad30 msgr2=0x7f00100fb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:16.586 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.600+0000 7f0018613700 1 -- 172.21.15.184:0/2002943819 shutdown_connections 2024-04-06T02:43:16.586 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:16.600+0000 7f0018613700 1 -- 172.21.15.184:0/2002943819 wait complete. 2024-04-06T02:43:16.586 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-04-06T02:43:17.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:17 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/2002943819' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T02:43:18.179 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T02:43:18.179 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T02:43:18.382 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:18.427 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:18.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:18 smithi105 ceph-mon[29228]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:43:19.828 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.842+0000 7fe29f126700 1 -- 172.21.15.184:0/527286455 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2980ff790 msgr2=0x7fe2980ffbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:19.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.842+0000 7fe29f126700 1 --2- 172.21.15.184:0/527286455 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2980ff790 0x7fe2980ffbb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe28c009910 tx=0x7fe28c009c20 comp rx=0 tx=0).stop 2024-04-06T02:43:19.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.842+0000 7fe29f126700 1 -- 172.21.15.184:0/527286455 shutdown_connections 2024-04-06T02:43:19.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.842+0000 7fe29f126700 1 --2- 172.21.15.184:0/527286455 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2980ff790 0x7fe2980ffbb0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:19.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.842+0000 7fe29f126700 1 -- 172.21.15.184:0/527286455 >> 172.21.15.184:0/527286455 conn(0x7fe2980fad30 msgr2=0x7fe2980fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:19.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.842+0000 7fe29f126700 1 -- 172.21.15.184:0/527286455 shutdown_connections 2024-04-06T02:43:19.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.842+0000 7fe29f126700 1 -- 172.21.15.184:0/527286455 wait complete. 2024-04-06T02:43:19.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.843+0000 7fe29f126700 1 Processor -- start 2024-04-06T02:43:19.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.843+0000 7fe29f126700 1 -- start start 2024-04-06T02:43:19.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.843+0000 7fe29f126700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2980ff790 0x7fe298194890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:19.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.843+0000 7fe29f126700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe298194dd0 con 0x7fe2980ff790 2024-04-06T02:43:19.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.843+0000 7fe29cec2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2980ff790 0x7fe298194890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:19.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.843+0000 7fe29cec2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2980ff790 0x7fe298194890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.184:54850/0 (socket says 172.21.15.184:54850) 2024-04-06T02:43:19.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.843+0000 7fe29cec2700 1 -- 172.21.15.184:0/4072665594 learned_addr learned my addr 172.21.15.184:0/4072665594 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-04-06T02:43:19.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.843+0000 7fe29cec2700 1 -- 172.21.15.184:0/4072665594 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28c0095f0 con 0x7fe2980ff790 2024-04-06T02:43:19.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.843+0000 7fe29cec2700 1 --2- 172.21.15.184:0/4072665594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2980ff790 0x7fe298194890 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe28c005b40 tx=0x7fe28c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:19.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.844+0000 7fe295ffb700 1 -- 172.21.15.184:0/4072665594 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe28c01d070 con 0x7fe2980ff790 2024-04-06T02:43:19.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.844+0000 7fe295ffb700 1 -- 172.21.15.184:0/4072665594 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe28c00be10 con 0x7fe2980ff790 2024-04-06T02:43:19.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.844+0000 7fe29f126700 1 -- 172.21.15.184:0/4072665594 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe298194fd0 con 0x7fe2980ff790 2024-04-06T02:43:19.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.844+0000 7fe295ffb700 1 -- 172.21.15.184:0/4072665594 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe28c00f460 con 0x7fe2980ff790 2024-04-06T02:43:19.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.844+0000 7fe29f126700 1 -- 172.21.15.184:0/4072665594 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe298195470 con 0x7fe2980ff790 2024-04-06T02:43:19.831 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.845+0000 7fe29f126700 1 -- 172.21.15.184:0/4072665594 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe29818e3b0 con 0x7fe2980ff790 2024-04-06T02:43:19.831 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.845+0000 7fe295ffb700 1 -- 172.21.15.184:0/4072665594 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fe28c0227e0 con 0x7fe2980ff790 2024-04-06T02:43:19.831 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.845+0000 7fe295ffb700 1 --2- 172.21.15.184:0/4072665594 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fe2880383c0 0x7fe28803a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:19.831 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.845+0000 7fe295ffb700 1 -- 172.21.15.184:0/4072665594 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe28c04c430 con 0x7fe2980ff790 2024-04-06T02:43:19.831 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.845+0000 7fe297fff700 1 --2- 172.21.15.184:0/4072665594 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fe2880383c0 0x7fe28803a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:19.831 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.846+0000 7fe297fff700 1 --2- 172.21.15.184:0/4072665594 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fe2880383c0 0x7fe28803a880 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe284006fd0 tx=0x7fe284006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:19.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:19.854+0000 7fe295ffb700 1 -- 172.21.15.184:0/4072665594 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe28c02bba0 con 0x7fe2980ff790 2024-04-06T02:43:20.102 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:20.117+0000 7fe29f126700 1 -- 172.21.15.184:0/4072665594 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe29802cc70 con 0x7fe2980ff790 2024-04-06T02:43:20.102 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:20.117+0000 7fe295ffb700 1 -- 172.21.15.184:0/4072665594 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fe28c027020 con 0x7fe2980ff790 2024-04-06T02:43:20.102 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:43:20.102 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"130df578-f3bf-11ee-b649-cb9ed24678a4","modified":"2024-04-06T02:41:29.716000Z","created":"2024-04-06T02:41:29.716000Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T02:43:20.104 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:20.119+0000 7fe29f126700 1 -- 172.21.15.184:0/4072665594 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fe2880383c0 msgr2=0x7fe28803a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:20.104 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:20.119+0000 7fe29f126700 1 --2- 172.21.15.184:0/4072665594 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fe2880383c0 0x7fe28803a880 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe284006fd0 tx=0x7fe284006e40 comp rx=0 tx=0).stop 2024-04-06T02:43:20.104 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:20.119+0000 7fe29f126700 1 -- 172.21.15.184:0/4072665594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2980ff790 msgr2=0x7fe298194890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:20.104 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:20.119+0000 7fe29f126700 1 --2- 172.21.15.184:0/4072665594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2980ff790 0x7fe298194890 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe28c005b40 tx=0x7fe28c004740 comp rx=0 tx=0).stop 2024-04-06T02:43:20.104 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:20.119+0000 7fe29f126700 1 -- 172.21.15.184:0/4072665594 shutdown_connections 2024-04-06T02:43:20.105 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:20.119+0000 7fe29f126700 1 --2- 172.21.15.184:0/4072665594 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fe2880383c0 0x7fe28803a880 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:20.105 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:20.119+0000 7fe29f126700 1 --2- 172.21.15.184:0/4072665594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2980ff790 0x7fe298194890 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:20.105 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:20.119+0000 7fe29f126700 1 -- 172.21.15.184:0/4072665594 >> 172.21.15.184:0/4072665594 conn(0x7fe2980fad30 msgr2=0x7fe2980fb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:20.105 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:20.119+0000 7fe29f126700 1 -- 172.21.15.184:0/4072665594 shutdown_connections 2024-04-06T02:43:20.105 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:20.119+0000 7fe29f126700 1 -- 172.21.15.184:0/4072665594 wait complete. 2024-04-06T02:43:20.105 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-04-06T02:43:20.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:20 smithi105 ceph-mon[29228]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:43:20.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:20 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/4072665594' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T02:43:21.713 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T02:43:21.713 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T02:43:21.916 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:21.962 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:22.575 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:22 smithi105 ceph-mon[29228]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:43:23.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.347+0000 7fa4f0247700 1 -- 172.21.15.184:0/4178527143 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4e80ff7a0 msgr2=0x7fa4e80ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:23.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.347+0000 7fa4f0247700 1 --2- 172.21.15.184:0/4178527143 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4e80ff7a0 0x7fa4e80ffbc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8009910 tx=0x7fa4d8009c20 comp rx=0 tx=0).stop 2024-04-06T02:43:23.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.347+0000 7fa4f0247700 1 -- 172.21.15.184:0/4178527143 shutdown_connections 2024-04-06T02:43:23.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.347+0000 7fa4f0247700 1 --2- 172.21.15.184:0/4178527143 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4e80ff7a0 0x7fa4e80ffbc0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:23.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.347+0000 7fa4f0247700 1 -- 172.21.15.184:0/4178527143 >> 172.21.15.184:0/4178527143 conn(0x7fa4e80fad20 msgr2=0x7fa4e80fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:23.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.347+0000 7fa4f0247700 1 -- 172.21.15.184:0/4178527143 shutdown_connections 2024-04-06T02:43:23.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.347+0000 7fa4f0247700 1 -- 172.21.15.184:0/4178527143 wait complete. 2024-04-06T02:43:23.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.348+0000 7fa4f0247700 1 Processor -- start 2024-04-06T02:43:23.334 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.348+0000 7fa4f0247700 1 -- start start 2024-04-06T02:43:23.334 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.348+0000 7fa4f0247700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4e80ff7a0 0x7fa4e81959d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:23.334 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.348+0000 7fa4f0247700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e8195f10 con 0x7fa4e80ff7a0 2024-04-06T02:43:23.334 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.348+0000 7fa4edfe3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4e80ff7a0 0x7fa4e81959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:23.334 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.349+0000 7fa4edfe3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4e80ff7a0 0x7fa4e81959d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.184:54876/0 (socket says 172.21.15.184:54876) 2024-04-06T02:43:23.334 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.349+0000 7fa4edfe3700 1 -- 172.21.15.184:0/2453208243 learned_addr learned my addr 172.21.15.184:0/2453208243 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-04-06T02:43:23.334 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.349+0000 7fa4edfe3700 1 -- 172.21.15.184:0/2453208243 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d80095f0 con 0x7fa4e80ff7a0 2024-04-06T02:43:23.334 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.349+0000 7fa4edfe3700 1 --2- 172.21.15.184:0/2453208243 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4e80ff7a0 0x7fa4e81959d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8005b40 tx=0x7fa4d8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:23.334 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.349+0000 7fa4deffd700 1 -- 172.21.15.184:0/2453208243 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4d801d070 con 0x7fa4e80ff7a0 2024-04-06T02:43:23.334 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.349+0000 7fa4f0247700 1 -- 172.21.15.184:0/2453208243 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e8196110 con 0x7fa4e80ff7a0 2024-04-06T02:43:23.334 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.349+0000 7fa4deffd700 1 -- 172.21.15.184:0/2453208243 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa4d800be10 con 0x7fa4e80ff7a0 2024-04-06T02:43:23.334 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.349+0000 7fa4deffd700 1 -- 172.21.15.184:0/2453208243 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4d800f460 con 0x7fa4e80ff7a0 2024-04-06T02:43:23.334 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.349+0000 7fa4f0247700 1 -- 172.21.15.184:0/2453208243 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4e81965b0 con 0x7fa4e80ff7a0 2024-04-06T02:43:23.335 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.350+0000 7fa4f0247700 1 -- 172.21.15.184:0/2453208243 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4e81055e0 con 0x7fa4e80ff7a0 2024-04-06T02:43:23.335 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.350+0000 7fa4deffd700 1 -- 172.21.15.184:0/2453208243 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fa4d80227e0 con 0x7fa4e80ff7a0 2024-04-06T02:43:23.335 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.350+0000 7fa4deffd700 1 --2- 172.21.15.184:0/2453208243 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fa4d40383c0 0x7fa4d403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:23.335 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.350+0000 7fa4deffd700 1 -- 172.21.15.184:0/2453208243 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa4d804c4e0 con 0x7fa4e80ff7a0 2024-04-06T02:43:23.336 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.351+0000 7fa4ed7e2700 1 --2- 172.21.15.184:0/2453208243 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fa4d40383c0 0x7fa4d403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:23.336 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.351+0000 7fa4ed7e2700 1 --2- 172.21.15.184:0/2453208243 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fa4d40383c0 0x7fa4d403a880 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa4e4006fd0 tx=0x7fa4e4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:23.339 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.356+0000 7fa4deffd700 1 -- 172.21.15.184:0/2453208243 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa4d802bba0 con 0x7fa4e80ff7a0 2024-04-06T02:43:23.607 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.624+0000 7fa4f0247700 1 -- 172.21.15.184:0/2453208243 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4e8199390 con 0x7fa4e80ff7a0 2024-04-06T02:43:23.608 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.624+0000 7fa4deffd700 1 -- 172.21.15.184:0/2453208243 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fa4d8027020 con 0x7fa4e80ff7a0 2024-04-06T02:43:23.608 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:43:23.608 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"130df578-f3bf-11ee-b649-cb9ed24678a4","modified":"2024-04-06T02:41:29.716000Z","created":"2024-04-06T02:41:29.716000Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T02:43:23.610 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.626+0000 7fa4f0247700 1 -- 172.21.15.184:0/2453208243 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fa4d40383c0 msgr2=0x7fa4d403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:23.610 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.626+0000 7fa4f0247700 1 --2- 172.21.15.184:0/2453208243 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fa4d40383c0 0x7fa4d403a880 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa4e4006fd0 tx=0x7fa4e4006e40 comp rx=0 tx=0).stop 2024-04-06T02:43:23.610 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.626+0000 7fa4f0247700 1 -- 172.21.15.184:0/2453208243 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4e80ff7a0 msgr2=0x7fa4e81959d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:23.610 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.626+0000 7fa4f0247700 1 --2- 172.21.15.184:0/2453208243 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4e80ff7a0 0x7fa4e81959d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8005b40 tx=0x7fa4d8004740 comp rx=0 tx=0).stop 2024-04-06T02:43:23.611 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.626+0000 7fa4f0247700 1 -- 172.21.15.184:0/2453208243 shutdown_connections 2024-04-06T02:43:23.611 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.626+0000 7fa4f0247700 1 --2- 172.21.15.184:0/2453208243 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fa4d40383c0 0x7fa4d403a880 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:23.611 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.626+0000 7fa4f0247700 1 --2- 172.21.15.184:0/2453208243 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4e80ff7a0 0x7fa4e81959d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:23.611 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.626+0000 7fa4f0247700 1 -- 172.21.15.184:0/2453208243 >> 172.21.15.184:0/2453208243 conn(0x7fa4e80fad20 msgr2=0x7fa4e80fba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:23.611 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.626+0000 7fa4f0247700 1 -- 172.21.15.184:0/2453208243 shutdown_connections 2024-04-06T02:43:23.611 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:23.626+0000 7fa4f0247700 1 -- 172.21.15.184:0/2453208243 wait complete. 2024-04-06T02:43:23.611 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-04-06T02:43:24.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:24 smithi105 ceph-mon[29228]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:43:24.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:24 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/2453208243' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T02:43:25.157 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T02:43:25.157 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T02:43:25.362 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:25.407 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:26.663 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:26 smithi105 ceph-mon[29228]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:43:26.767 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.783+0000 7f9cc43d6700 1 -- 172.21.15.184:0/2954625272 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9cbc1008b0 msgr2=0x7f9cbc100cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:26.767 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.783+0000 7f9cc43d6700 1 --2- 172.21.15.184:0/2954625272 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9cbc1008b0 0x7f9cbc100cd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9cb8009910 tx=0x7f9cb8009c20 comp rx=0 tx=0).stop 2024-04-06T02:43:26.767 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.783+0000 7f9cc43d6700 1 -- 172.21.15.184:0/2954625272 shutdown_connections 2024-04-06T02:43:26.767 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.783+0000 7f9cc43d6700 1 --2- 172.21.15.184:0/2954625272 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9cbc1008b0 0x7f9cbc100cd0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:26.768 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.783+0000 7f9cc43d6700 1 -- 172.21.15.184:0/2954625272 >> 172.21.15.184:0/2954625272 conn(0x7f9cbc0fbe30 msgr2=0x7f9cbc0fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:26.768 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.783+0000 7f9cc43d6700 1 -- 172.21.15.184:0/2954625272 shutdown_connections 2024-04-06T02:43:26.768 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.783+0000 7f9cc43d6700 1 -- 172.21.15.184:0/2954625272 wait complete. 2024-04-06T02:43:26.769 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.784+0000 7f9cc43d6700 1 Processor -- start 2024-04-06T02:43:26.769 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.784+0000 7f9cc43d6700 1 -- start start 2024-04-06T02:43:26.769 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.784+0000 7f9cc43d6700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9cbc1008b0 0x7f9cbc1959e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:26.769 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.784+0000 7f9cc43d6700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cbc195f20 con 0x7f9cbc1008b0 2024-04-06T02:43:26.769 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.784+0000 7f9cc2172700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9cbc1008b0 0x7f9cbc1959e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:26.769 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.784+0000 7f9cc2172700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9cbc1008b0 0x7f9cbc1959e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.184:54892/0 (socket says 172.21.15.184:54892) 2024-04-06T02:43:26.769 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.784+0000 7f9cc2172700 1 -- 172.21.15.184:0/2847370974 learned_addr learned my addr 172.21.15.184:0/2847370974 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-04-06T02:43:26.769 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.785+0000 7f9cc2172700 1 -- 172.21.15.184:0/2847370974 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb80095f0 con 0x7f9cbc1008b0 2024-04-06T02:43:26.769 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.785+0000 7f9cc2172700 1 --2- 172.21.15.184:0/2847370974 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9cbc1008b0 0x7f9cbc1959e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9cb8005b40 tx=0x7f9cb8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:26.770 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.785+0000 7f9cb37fe700 1 -- 172.21.15.184:0/2847370974 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cb801d070 con 0x7f9cbc1008b0 2024-04-06T02:43:26.770 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.785+0000 7f9cc43d6700 1 -- 172.21.15.184:0/2847370974 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cbc196120 con 0x7f9cbc1008b0 2024-04-06T02:43:26.770 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.785+0000 7f9cb37fe700 1 -- 172.21.15.184:0/2847370974 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9cb800be10 con 0x7f9cbc1008b0 2024-04-06T02:43:26.770 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.785+0000 7f9cb37fe700 1 -- 172.21.15.184:0/2847370974 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cb800f460 con 0x7f9cbc1008b0 2024-04-06T02:43:26.770 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.785+0000 7f9cc43d6700 1 -- 172.21.15.184:0/2847370974 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cbc1965c0 con 0x7f9cbc1008b0 2024-04-06T02:43:26.771 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.786+0000 7f9cc43d6700 1 -- 172.21.15.184:0/2847370974 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cbc18f4d0 con 0x7f9cbc1008b0 2024-04-06T02:43:26.771 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.786+0000 7f9cb37fe700 1 -- 172.21.15.184:0/2847370974 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f9cb80227e0 con 0x7f9cbc1008b0 2024-04-06T02:43:26.771 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.786+0000 7f9cb37fe700 1 --2- 172.21.15.184:0/2847370974 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f9ca80383c0 0x7f9ca803a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:26.771 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.786+0000 7f9cb37fe700 1 -- 172.21.15.184:0/2847370974 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9cb804c4a0 con 0x7f9cbc1008b0 2024-04-06T02:43:26.771 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.787+0000 7f9cc1971700 1 --2- 172.21.15.184:0/2847370974 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f9ca80383c0 0x7f9ca803a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:26.771 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.787+0000 7f9cc1971700 1 --2- 172.21.15.184:0/2847370974 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f9ca80383c0 0x7f9ca803a880 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9cac006fd0 tx=0x7f9cac006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:26.780 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:26.797+0000 7f9cb37fe700 1 -- 172.21.15.184:0/2847370974 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f9cb802bba0 con 0x7f9cbc1008b0 2024-04-06T02:43:27.045 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:27.062+0000 7f9cc43d6700 1 -- 172.21.15.184:0/2847370974 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9cbc02cc70 con 0x7f9cbc1008b0 2024-04-06T02:43:27.046 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:27.062+0000 7f9cb37fe700 1 -- 172.21.15.184:0/2847370974 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f9cb8027020 con 0x7f9cbc1008b0 2024-04-06T02:43:27.046 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:43:27.046 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"130df578-f3bf-11ee-b649-cb9ed24678a4","modified":"2024-04-06T02:41:29.716000Z","created":"2024-04-06T02:41:29.716000Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T02:43:27.048 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:27.064+0000 7f9cc43d6700 1 -- 172.21.15.184:0/2847370974 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f9ca80383c0 msgr2=0x7f9ca803a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:27.048 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:27.064+0000 7f9cc43d6700 1 --2- 172.21.15.184:0/2847370974 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f9ca80383c0 0x7f9ca803a880 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9cac006fd0 tx=0x7f9cac006e40 comp rx=0 tx=0).stop 2024-04-06T02:43:27.048 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:27.064+0000 7f9cc43d6700 1 -- 172.21.15.184:0/2847370974 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9cbc1008b0 msgr2=0x7f9cbc1959e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:27.048 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:27.064+0000 7f9cc43d6700 1 --2- 172.21.15.184:0/2847370974 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9cbc1008b0 0x7f9cbc1959e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9cb8005b40 tx=0x7f9cb8004740 comp rx=0 tx=0).stop 2024-04-06T02:43:27.048 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:27.064+0000 7f9cc43d6700 1 -- 172.21.15.184:0/2847370974 shutdown_connections 2024-04-06T02:43:27.048 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:27.064+0000 7f9cc43d6700 1 --2- 172.21.15.184:0/2847370974 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f9ca80383c0 0x7f9ca803a880 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:27.048 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:27.064+0000 7f9cc43d6700 1 --2- 172.21.15.184:0/2847370974 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9cbc1008b0 0x7f9cbc1959e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:27.049 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:27.064+0000 7f9cc43d6700 1 -- 172.21.15.184:0/2847370974 >> 172.21.15.184:0/2847370974 conn(0x7f9cbc0fbe30 msgr2=0x7f9cbc0fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:27.049 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:27.064+0000 7f9cc43d6700 1 -- 172.21.15.184:0/2847370974 shutdown_connections 2024-04-06T02:43:27.049 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:27.064+0000 7f9cc43d6700 1 -- 172.21.15.184:0/2847370974 wait complete. 2024-04-06T02:43:27.049 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-04-06T02:43:27.551 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:27 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/2847370974' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T02:43:28.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:28 smithi105 ceph-mon[29228]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:43:28.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:28 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:28.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:28 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:28.304 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:28 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:28.304 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:28 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:28.304 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:28 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:28.304 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:28 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:28.304 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:28 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:28.304 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:28 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:28.570 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T02:43:28.571 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T02:43:28.773 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:28.818 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:29.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:29 smithi105 ceph-mon[29228]: Deploying daemon prometheus.smithi105 on smithi105 2024-04-06T02:43:30.205 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.220+0000 7f1f4140a700 1 -- 172.21.15.184:0/289272675 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f3c0fef20 msgr2=0x7f1f3c0ff340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:30.205 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.220+0000 7f1f4140a700 1 --2- 172.21.15.184:0/289272675 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f3c0fef20 0x7f1f3c0ff340 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1f30009910 tx=0x7f1f30009c20 comp rx=0 tx=0).stop 2024-04-06T02:43:30.205 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.220+0000 7f1f4140a700 1 -- 172.21.15.184:0/289272675 shutdown_connections 2024-04-06T02:43:30.206 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.220+0000 7f1f4140a700 1 --2- 172.21.15.184:0/289272675 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f3c0fef20 0x7f1f3c0ff340 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:30.206 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.220+0000 7f1f4140a700 1 -- 172.21.15.184:0/289272675 >> 172.21.15.184:0/289272675 conn(0x7f1f3c0fa4a0 msgr2=0x7f1f3c0fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:30.206 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.220+0000 7f1f4140a700 1 -- 172.21.15.184:0/289272675 shutdown_connections 2024-04-06T02:43:30.206 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.220+0000 7f1f4140a700 1 -- 172.21.15.184:0/289272675 wait complete. 2024-04-06T02:43:30.206 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.221+0000 7f1f4140a700 1 Processor -- start 2024-04-06T02:43:30.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.221+0000 7f1f4140a700 1 -- start start 2024-04-06T02:43:30.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.221+0000 7f1f4140a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f3c0fef20 0x7f1f3c197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:30.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.221+0000 7f1f4140a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c198090 con 0x7f1f3c0fef20 2024-04-06T02:43:30.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.221+0000 7f1f3affd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f3c0fef20 0x7f1f3c197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:30.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.221+0000 7f1f3affd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f3c0fef20 0x7f1f3c197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.184:49084/0 (socket says 172.21.15.184:49084) 2024-04-06T02:43:30.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.221+0000 7f1f3affd700 1 -- 172.21.15.184:0/2088910914 learned_addr learned my addr 172.21.15.184:0/2088910914 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-04-06T02:43:30.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.222+0000 7f1f3affd700 1 -- 172.21.15.184:0/2088910914 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f300095f0 con 0x7f1f3c0fef20 2024-04-06T02:43:30.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.222+0000 7f1f3affd700 1 --2- 172.21.15.184:0/2088910914 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f3c0fef20 0x7f1f3c197b50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1f300092e0 tx=0x7f1f30009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:30.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.222+0000 7f1f23fff700 1 -- 172.21.15.184:0/2088910914 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f3001d070 con 0x7f1f3c0fef20 2024-04-06T02:43:30.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.222+0000 7f1f23fff700 1 -- 172.21.15.184:0/2088910914 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1f3000be10 con 0x7f1f3c0fef20 2024-04-06T02:43:30.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.222+0000 7f1f4140a700 1 -- 172.21.15.184:0/2088910914 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f3c198290 con 0x7f1f3c0fef20 2024-04-06T02:43:30.209 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.222+0000 7f1f23fff700 1 -- 172.21.15.184:0/2088910914 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f3000f460 con 0x7f1f3c0fef20 2024-04-06T02:43:30.209 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.222+0000 7f1f4140a700 1 -- 172.21.15.184:0/2088910914 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f3c198730 con 0x7f1f3c0fef20 2024-04-06T02:43:30.209 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.223+0000 7f1f23fff700 1 -- 172.21.15.184:0/2088910914 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f1f3000b960 con 0x7f1f3c0fef20 2024-04-06T02:43:30.209 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.223+0000 7f1f4140a700 1 -- 172.21.15.184:0/2088910914 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f3c191fd0 con 0x7f1f3c0fef20 2024-04-06T02:43:30.209 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.223+0000 7f1f23fff700 1 --2- 172.21.15.184:0/2088910914 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f1f240383c0 0x7f1f2403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:30.209 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.223+0000 7f1f23fff700 1 -- 172.21.15.184:0/2088910914 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1f3004c570 con 0x7f1f3c0fef20 2024-04-06T02:43:30.209 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.223+0000 7f1f3a7fc700 1 --2- 172.21.15.184:0/2088910914 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f1f240383c0 0x7f1f2403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:30.209 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.224+0000 7f1f3a7fc700 1 --2- 172.21.15.184:0/2088910914 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f1f240383c0 0x7f1f2403a880 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c006fd0 tx=0x7f1f2c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:30.213 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.228+0000 7f1f23fff700 1 -- 172.21.15.184:0/2088910914 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1f30005c00 con 0x7f1f3c0fef20 2024-04-06T02:43:30.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.492+0000 7f1f4140a700 1 -- 172.21.15.184:0/2088910914 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f3c02cc70 con 0x7f1f3c0fef20 2024-04-06T02:43:30.478 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.493+0000 7f1f23fff700 1 -- 172.21.15.184:0/2088910914 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f1f30027070 con 0x7f1f3c0fef20 2024-04-06T02:43:30.478 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:43:30.478 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"130df578-f3bf-11ee-b649-cb9ed24678a4","modified":"2024-04-06T02:41:29.716000Z","created":"2024-04-06T02:41:29.716000Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T02:43:30.480 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.495+0000 7f1f4140a700 1 -- 172.21.15.184:0/2088910914 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f1f240383c0 msgr2=0x7f1f2403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:30.480 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.495+0000 7f1f4140a700 1 --2- 172.21.15.184:0/2088910914 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f1f240383c0 0x7f1f2403a880 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c006fd0 tx=0x7f1f2c006e40 comp rx=0 tx=0).stop 2024-04-06T02:43:30.480 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.495+0000 7f1f4140a700 1 -- 172.21.15.184:0/2088910914 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f3c0fef20 msgr2=0x7f1f3c197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:30.480 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.495+0000 7f1f4140a700 1 --2- 172.21.15.184:0/2088910914 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f3c0fef20 0x7f1f3c197b50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1f300092e0 tx=0x7f1f30009e60 comp rx=0 tx=0).stop 2024-04-06T02:43:30.480 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.495+0000 7f1f4140a700 1 -- 172.21.15.184:0/2088910914 shutdown_connections 2024-04-06T02:43:30.481 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.495+0000 7f1f4140a700 1 --2- 172.21.15.184:0/2088910914 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f1f240383c0 0x7f1f2403a880 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:30.481 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.495+0000 7f1f4140a700 1 --2- 172.21.15.184:0/2088910914 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f3c0fef20 0x7f1f3c197b50 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:30.481 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.495+0000 7f1f4140a700 1 -- 172.21.15.184:0/2088910914 >> 172.21.15.184:0/2088910914 conn(0x7f1f3c0fa4a0 msgr2=0x7f1f3c0fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:30.481 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.495+0000 7f1f4140a700 1 -- 172.21.15.184:0/2088910914 shutdown_connections 2024-04-06T02:43:30.481 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:30.495+0000 7f1f4140a700 1 -- 172.21.15.184:0/2088910914 wait complete. 2024-04-06T02:43:30.481 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-04-06T02:43:30.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:30 smithi105 ceph-mon[29228]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:43:30.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:30 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:31.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:31 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/2088910914' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T02:43:32.041 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T02:43:32.041 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T02:43:32.242 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:32.287 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:32.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:32 smithi105 ceph-mon[29228]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:43:33.642 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.656+0000 7fb132e7e700 1 -- 172.21.15.184:0/2304607156 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb12c0ff790 msgr2=0x7fb12c0ffbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:33.642 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.656+0000 7fb132e7e700 1 --2- 172.21.15.184:0/2304607156 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb12c0ff790 0x7fb12c0ffbb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb120009910 tx=0x7fb120009c20 comp rx=0 tx=0).stop 2024-04-06T02:43:33.642 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.656+0000 7fb132e7e700 1 -- 172.21.15.184:0/2304607156 shutdown_connections 2024-04-06T02:43:33.643 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.656+0000 7fb132e7e700 1 --2- 172.21.15.184:0/2304607156 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb12c0ff790 0x7fb12c0ffbb0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:33.643 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.656+0000 7fb132e7e700 1 -- 172.21.15.184:0/2304607156 >> 172.21.15.184:0/2304607156 conn(0x7fb12c0fad30 msgr2=0x7fb12c0fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:33.643 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.656+0000 7fb132e7e700 1 -- 172.21.15.184:0/2304607156 shutdown_connections 2024-04-06T02:43:33.643 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.656+0000 7fb132e7e700 1 -- 172.21.15.184:0/2304607156 wait complete. 2024-04-06T02:43:33.643 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.656+0000 7fb132e7e700 1 Processor -- start 2024-04-06T02:43:33.643 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.656+0000 7fb132e7e700 1 -- start start 2024-04-06T02:43:33.643 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.656+0000 7fb132e7e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb12c0ff790 0x7fb12c0fccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:33.643 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.656+0000 7fb132e7e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12c0fd210 con 0x7fb12c0ff790 2024-04-06T02:43:33.643 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.657+0000 7fb130c1a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb12c0ff790 0x7fb12c0fccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:33.643 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.657+0000 7fb130c1a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb12c0ff790 0x7fb12c0fccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.184:49114/0 (socket says 172.21.15.184:49114) 2024-04-06T02:43:33.643 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.657+0000 7fb130c1a700 1 -- 172.21.15.184:0/2284674981 learned_addr learned my addr 172.21.15.184:0/2284674981 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-04-06T02:43:33.643 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.657+0000 7fb130c1a700 1 -- 172.21.15.184:0/2284674981 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1200095f0 con 0x7fb12c0ff790 2024-04-06T02:43:33.643 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.657+0000 7fb130c1a700 1 --2- 172.21.15.184:0/2284674981 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb12c0ff790 0x7fb12c0fccd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb120005b40 tx=0x7fb120004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:33.644 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.657+0000 7fb129ffb700 1 -- 172.21.15.184:0/2284674981 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb12001d070 con 0x7fb12c0ff790 2024-04-06T02:43:33.644 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.657+0000 7fb129ffb700 1 -- 172.21.15.184:0/2284674981 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb12000be10 con 0x7fb12c0ff790 2024-04-06T02:43:33.644 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.657+0000 7fb132e7e700 1 -- 172.21.15.184:0/2284674981 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb12c0fb320 con 0x7fb12c0ff790 2024-04-06T02:43:33.644 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.657+0000 7fb129ffb700 1 -- 172.21.15.184:0/2284674981 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb12000f460 con 0x7fb12c0ff790 2024-04-06T02:43:33.644 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.657+0000 7fb132e7e700 1 -- 172.21.15.184:0/2284674981 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb12c0fb7c0 con 0x7fb12c0ff790 2024-04-06T02:43:33.645 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.658+0000 7fb129ffb700 1 -- 172.21.15.184:0/2284674981 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fb1200227e0 con 0x7fb12c0ff790 2024-04-06T02:43:33.645 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.658+0000 7fb132e7e700 1 -- 172.21.15.184:0/2284674981 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb12c0623c0 con 0x7fb12c0ff790 2024-04-06T02:43:33.645 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.658+0000 7fb129ffb700 1 --2- 172.21.15.184:0/2284674981 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fb11c0383c0 0x7fb11c03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:33.645 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.658+0000 7fb129ffb700 1 -- 172.21.15.184:0/2284674981 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb12004c560 con 0x7fb12c0ff790 2024-04-06T02:43:33.645 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.659+0000 7fb12bfff700 1 --2- 172.21.15.184:0/2284674981 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fb11c0383c0 0x7fb11c03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:33.645 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.659+0000 7fb12bfff700 1 --2- 172.21.15.184:0/2284674981 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fb11c0383c0 0x7fb11c03a880 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb118006fd0 tx=0x7fb118006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:33.648 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.664+0000 7fb129ffb700 1 -- 172.21.15.184:0/2284674981 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb12002bba0 con 0x7fb12c0ff790 2024-04-06T02:43:33.909 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.924+0000 7fb132e7e700 1 -- 172.21.15.184:0/2284674981 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb12c18eda0 con 0x7fb12c0ff790 2024-04-06T02:43:33.909 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.924+0000 7fb129ffb700 1 -- 172.21.15.184:0/2284674981 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fb120027020 con 0x7fb12c0ff790 2024-04-06T02:43:33.909 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:43:33.909 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"130df578-f3bf-11ee-b649-cb9ed24678a4","modified":"2024-04-06T02:41:29.716000Z","created":"2024-04-06T02:41:29.716000Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T02:43:33.911 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.926+0000 7fb132e7e700 1 -- 172.21.15.184:0/2284674981 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fb11c0383c0 msgr2=0x7fb11c03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:33.911 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.926+0000 7fb132e7e700 1 --2- 172.21.15.184:0/2284674981 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fb11c0383c0 0x7fb11c03a880 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb118006fd0 tx=0x7fb118006e40 comp rx=0 tx=0).stop 2024-04-06T02:43:33.911 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.926+0000 7fb132e7e700 1 -- 172.21.15.184:0/2284674981 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb12c0ff790 msgr2=0x7fb12c0fccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:33.911 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.926+0000 7fb132e7e700 1 --2- 172.21.15.184:0/2284674981 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb12c0ff790 0x7fb12c0fccd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb120005b40 tx=0x7fb120004740 comp rx=0 tx=0).stop 2024-04-06T02:43:33.911 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.926+0000 7fb132e7e700 1 -- 172.21.15.184:0/2284674981 shutdown_connections 2024-04-06T02:43:33.911 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.926+0000 7fb132e7e700 1 --2- 172.21.15.184:0/2284674981 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fb11c0383c0 0x7fb11c03a880 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:33.912 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.926+0000 7fb132e7e700 1 --2- 172.21.15.184:0/2284674981 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb12c0ff790 0x7fb12c0fccd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:33.912 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.926+0000 7fb132e7e700 1 -- 172.21.15.184:0/2284674981 >> 172.21.15.184:0/2284674981 conn(0x7fb12c0fad30 msgr2=0x7fb12c069120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:33.912 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.926+0000 7fb132e7e700 1 -- 172.21.15.184:0/2284674981 shutdown_connections 2024-04-06T02:43:33.912 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:33.927+0000 7fb132e7e700 1 -- 172.21.15.184:0/2284674981 wait complete. 2024-04-06T02:43:33.912 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-04-06T02:43:34.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:34 smithi105 ceph-mon[29228]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:43:34.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:34 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/2284674981' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T02:43:35.452 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T02:43:35.452 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T02:43:35.654 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:35.700 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:36.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:36 smithi105 ceph-mon[29228]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:43:37.178 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.193+0000 7fceb5237700 1 -- 172.21.15.184:0/3887912514 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fceb0101320 msgr2=0x7fceb0103710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:37.178 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.193+0000 7fceb5237700 1 --2- 172.21.15.184:0/3887912514 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fceb0101320 0x7fceb0103710 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcea0009910 tx=0x7fcea0009c20 comp rx=0 tx=0).stop 2024-04-06T02:43:37.178 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.193+0000 7fceb5237700 1 -- 172.21.15.184:0/3887912514 shutdown_connections 2024-04-06T02:43:37.178 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.193+0000 7fceb5237700 1 --2- 172.21.15.184:0/3887912514 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fceb0101320 0x7fceb0103710 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:37.178 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.193+0000 7fceb5237700 1 -- 172.21.15.184:0/3887912514 >> 172.21.15.184:0/3887912514 conn(0x7fceb00fad30 msgr2=0x7fceb00fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:37.178 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.193+0000 7fceb5237700 1 -- 172.21.15.184:0/3887912514 shutdown_connections 2024-04-06T02:43:37.178 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.193+0000 7fceb5237700 1 -- 172.21.15.184:0/3887912514 wait complete. 2024-04-06T02:43:37.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.194+0000 7fceb5237700 1 Processor -- start 2024-04-06T02:43:37.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.194+0000 7fceb5237700 1 -- start start 2024-04-06T02:43:37.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.194+0000 7fceb5237700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fceb0101320 0x7fceb0195940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:37.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.194+0000 7fceb5237700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb0195e80 con 0x7fceb0101320 2024-04-06T02:43:37.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.194+0000 7fceaed9d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fceb0101320 0x7fceb0195940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:37.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.194+0000 7fceaed9d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fceb0101320 0x7fceb0195940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.184:49126/0 (socket says 172.21.15.184:49126) 2024-04-06T02:43:37.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.194+0000 7fceaed9d700 1 -- 172.21.15.184:0/3832507437 learned_addr learned my addr 172.21.15.184:0/3832507437 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-04-06T02:43:37.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.195+0000 7fceaed9d700 1 -- 172.21.15.184:0/3832507437 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea00095f0 con 0x7fceb0101320 2024-04-06T02:43:37.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.195+0000 7fceaed9d700 1 --2- 172.21.15.184:0/3832507437 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fceb0101320 0x7fceb0195940 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcea00092e0 tx=0x7fcea0009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:37.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.195+0000 7fce97fff700 1 -- 172.21.15.184:0/3832507437 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcea001d070 con 0x7fceb0101320 2024-04-06T02:43:37.181 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.195+0000 7fce97fff700 1 -- 172.21.15.184:0/3832507437 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcea000be10 con 0x7fceb0101320 2024-04-06T02:43:37.181 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.195+0000 7fceb5237700 1 -- 172.21.15.184:0/3832507437 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceb0196080 con 0x7fceb0101320 2024-04-06T02:43:37.181 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.195+0000 7fce97fff700 1 -- 172.21.15.184:0/3832507437 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcea000f460 con 0x7fceb0101320 2024-04-06T02:43:37.181 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.195+0000 7fceb5237700 1 -- 172.21.15.184:0/3832507437 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceb0196520 con 0x7fceb0101320 2024-04-06T02:43:37.182 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.196+0000 7fce97fff700 1 -- 172.21.15.184:0/3832507437 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fcea000f6d0 con 0x7fceb0101320 2024-04-06T02:43:37.182 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.196+0000 7fceb5237700 1 -- 172.21.15.184:0/3832507437 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceb018f4b0 con 0x7fceb0101320 2024-04-06T02:43:37.182 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.196+0000 7fce97fff700 1 --2- 172.21.15.184:0/3832507437 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fce980383c0 0x7fce9803a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:37.182 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.196+0000 7fce97fff700 1 -- 172.21.15.184:0/3832507437 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcea004c560 con 0x7fceb0101320 2024-04-06T02:43:37.182 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.196+0000 7fceae59c700 1 --2- 172.21.15.184:0/3832507437 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fce980383c0 0x7fce9803a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:37.182 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.197+0000 7fceae59c700 1 --2- 172.21.15.184:0/3832507437 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fce980383c0 0x7fce9803a880 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcea4006fd0 tx=0x7fcea4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:37.186 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.201+0000 7fce97fff700 1 -- 172.21.15.184:0/3832507437 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fcea0005c00 con 0x7fceb0101320 2024-04-06T02:43:37.453 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.469+0000 7fceb5237700 1 -- 172.21.15.184:0/3832507437 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fceb002cc70 con 0x7fceb0101320 2024-04-06T02:43:37.454 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.469+0000 7fce97fff700 1 -- 172.21.15.184:0/3832507437 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fcea0027030 con 0x7fceb0101320 2024-04-06T02:43:37.454 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:43:37.454 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"130df578-f3bf-11ee-b649-cb9ed24678a4","modified":"2024-04-06T02:41:29.716000Z","created":"2024-04-06T02:41:29.716000Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T02:43:37.456 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.471+0000 7fceb5237700 1 -- 172.21.15.184:0/3832507437 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fce980383c0 msgr2=0x7fce9803a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:37.456 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.471+0000 7fceb5237700 1 --2- 172.21.15.184:0/3832507437 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fce980383c0 0x7fce9803a880 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcea4006fd0 tx=0x7fcea4006e40 comp rx=0 tx=0).stop 2024-04-06T02:43:37.456 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.471+0000 7fceb5237700 1 -- 172.21.15.184:0/3832507437 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fceb0101320 msgr2=0x7fceb0195940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:37.456 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.471+0000 7fceb5237700 1 --2- 172.21.15.184:0/3832507437 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fceb0101320 0x7fceb0195940 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcea00092e0 tx=0x7fcea0009e60 comp rx=0 tx=0).stop 2024-04-06T02:43:37.456 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.471+0000 7fceb5237700 1 -- 172.21.15.184:0/3832507437 shutdown_connections 2024-04-06T02:43:37.457 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.471+0000 7fceb5237700 1 --2- 172.21.15.184:0/3832507437 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fce980383c0 0x7fce9803a880 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:37.457 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.471+0000 7fceb5237700 1 --2- 172.21.15.184:0/3832507437 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fceb0101320 0x7fceb0195940 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:37.457 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.471+0000 7fceb5237700 1 -- 172.21.15.184:0/3832507437 >> 172.21.15.184:0/3832507437 conn(0x7fceb00fad30 msgr2=0x7fceb00fb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:37.457 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.472+0000 7fceb5237700 1 -- 172.21.15.184:0/3832507437 shutdown_connections 2024-04-06T02:43:37.457 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:37.472+0000 7fceb5237700 1 -- 172.21.15.184:0/3832507437 wait complete. 2024-04-06T02:43:37.457 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-04-06T02:43:38.413 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:38 smithi105 ceph-mon[29228]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:43:38.413 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:38 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/3832507437' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T02:43:39.051 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T02:43:39.051 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T02:43:39.252 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:39.300 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:40.556 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.570+0000 7f5ccc620700 1 -- 172.21.15.184:0/586284105 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cc40ff7a0 msgr2=0x7f5cc40ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:40.556 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.570+0000 7f5ccc620700 1 --2- 172.21.15.184:0/586284105 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cc40ff7a0 0x7f5cc40ffbc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4009910 tx=0x7f5cb4009c20 comp rx=0 tx=0).stop 2024-04-06T02:43:40.556 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.571+0000 7f5ccc620700 1 -- 172.21.15.184:0/586284105 shutdown_connections 2024-04-06T02:43:40.556 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.571+0000 7f5ccc620700 1 --2- 172.21.15.184:0/586284105 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cc40ff7a0 0x7f5cc40ffbc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:40.557 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.571+0000 7f5ccc620700 1 -- 172.21.15.184:0/586284105 >> 172.21.15.184:0/586284105 conn(0x7f5cc40fad20 msgr2=0x7f5cc40fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:40.557 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.571+0000 7f5ccc620700 1 -- 172.21.15.184:0/586284105 shutdown_connections 2024-04-06T02:43:40.557 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.571+0000 7f5ccc620700 1 -- 172.21.15.184:0/586284105 wait complete. 2024-04-06T02:43:40.557 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.571+0000 7f5ccc620700 1 Processor -- start 2024-04-06T02:43:40.557 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.571+0000 7f5ccc620700 1 -- start start 2024-04-06T02:43:40.557 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.571+0000 7f5ccc620700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cc40ff7a0 0x7f5cc41948a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:40.557 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.571+0000 7f5ccc620700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc4194de0 con 0x7f5cc40ff7a0 2024-04-06T02:43:40.557 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.572+0000 7f5cca3bc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cc40ff7a0 0x7f5cc41948a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:40.557 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.572+0000 7f5cca3bc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cc40ff7a0 0x7f5cc41948a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.184:48958/0 (socket says 172.21.15.184:48958) 2024-04-06T02:43:40.557 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.572+0000 7f5cca3bc700 1 -- 172.21.15.184:0/1074382790 learned_addr learned my addr 172.21.15.184:0/1074382790 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-04-06T02:43:40.559 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.572+0000 7f5cca3bc700 1 -- 172.21.15.184:0/1074382790 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb40095f0 con 0x7f5cc40ff7a0 2024-04-06T02:43:40.559 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.572+0000 7f5cca3bc700 1 --2- 172.21.15.184:0/1074382790 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cc40ff7a0 0x7f5cc41948a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4005b40 tx=0x7f5cb4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:40.559 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.572+0000 7f5cbb7fe700 1 -- 172.21.15.184:0/1074382790 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cb401f070 con 0x7f5cc40ff7a0 2024-04-06T02:43:40.559 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.572+0000 7f5ccc620700 1 -- 172.21.15.184:0/1074382790 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cc4194fe0 con 0x7f5cc40ff7a0 2024-04-06T02:43:40.559 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.572+0000 7f5cbb7fe700 1 -- 172.21.15.184:0/1074382790 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5cb400be10 con 0x7f5cc40ff7a0 2024-04-06T02:43:40.559 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.572+0000 7f5cbb7fe700 1 -- 172.21.15.184:0/1074382790 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cb4024620 con 0x7f5cc40ff7a0 2024-04-06T02:43:40.559 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.572+0000 7f5ccc620700 1 -- 172.21.15.184:0/1074382790 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cc4195480 con 0x7f5cc40ff7a0 2024-04-06T02:43:40.559 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.573+0000 7f5ccc620700 1 -- 172.21.15.184:0/1074382790 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cc41055e0 con 0x7f5cc40ff7a0 2024-04-06T02:43:40.559 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.573+0000 7f5cbb7fe700 1 -- 172.21.15.184:0/1074382790 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f5cb40199c0 con 0x7f5cc40ff7a0 2024-04-06T02:43:40.559 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.573+0000 7f5cbb7fe700 1 --2- 172.21.15.184:0/1074382790 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f5cb00383c0 0x7f5cb003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:40.559 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.574+0000 7f5cbb7fe700 1 -- 172.21.15.184:0/1074382790 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5cb404e640 con 0x7f5cc40ff7a0 2024-04-06T02:43:40.559 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.574+0000 7f5cc9bbb700 1 --2- 172.21.15.184:0/1074382790 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f5cb00383c0 0x7f5cb003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:40.560 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.574+0000 7f5cc9bbb700 1 --2- 172.21.15.184:0/1074382790 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f5cb00383c0 0x7f5cb003a880 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0006fd0 tx=0x7f5cc0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:40.563 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.578+0000 7f5cbb7fe700 1 -- 172.21.15.184:0/1074382790 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5cb402c540 con 0x7f5cc40ff7a0 2024-04-06T02:43:40.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:40 smithi105 ceph-mon[29228]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:43:40.827 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.842+0000 7f5ccc620700 1 -- 172.21.15.184:0/1074382790 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5cc41057d0 con 0x7f5cc40ff7a0 2024-04-06T02:43:40.827 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.843+0000 7f5cbb7fe700 1 -- 172.21.15.184:0/1074382790 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f5cb4029020 con 0x7f5cc40ff7a0 2024-04-06T02:43:40.828 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:43:40.828 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"130df578-f3bf-11ee-b649-cb9ed24678a4","modified":"2024-04-06T02:41:29.716000Z","created":"2024-04-06T02:41:29.716000Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T02:43:40.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.845+0000 7f5ccc620700 1 -- 172.21.15.184:0/1074382790 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f5cb00383c0 msgr2=0x7f5cb003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:40.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.845+0000 7f5ccc620700 1 --2- 172.21.15.184:0/1074382790 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f5cb00383c0 0x7f5cb003a880 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0006fd0 tx=0x7f5cc0006e40 comp rx=0 tx=0).stop 2024-04-06T02:43:40.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.845+0000 7f5ccc620700 1 -- 172.21.15.184:0/1074382790 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cc40ff7a0 msgr2=0x7f5cc41948a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:40.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.845+0000 7f5ccc620700 1 --2- 172.21.15.184:0/1074382790 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cc40ff7a0 0x7f5cc41948a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4005b40 tx=0x7f5cb4004740 comp rx=0 tx=0).stop 2024-04-06T02:43:40.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.845+0000 7f5ccc620700 1 -- 172.21.15.184:0/1074382790 shutdown_connections 2024-04-06T02:43:40.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.845+0000 7f5ccc620700 1 --2- 172.21.15.184:0/1074382790 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f5cb00383c0 0x7f5cb003a880 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:40.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.845+0000 7f5ccc620700 1 --2- 172.21.15.184:0/1074382790 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cc40ff7a0 0x7f5cc41948a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:40.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.845+0000 7f5ccc620700 1 -- 172.21.15.184:0/1074382790 >> 172.21.15.184:0/1074382790 conn(0x7f5cc40fad20 msgr2=0x7f5cc40fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:40.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.845+0000 7f5ccc620700 1 -- 172.21.15.184:0/1074382790 shutdown_connections 2024-04-06T02:43:40.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:40.845+0000 7f5ccc620700 1 -- 172.21.15.184:0/1074382790 wait complete. 2024-04-06T02:43:40.831 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-04-06T02:43:41.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:41 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/1074382790' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T02:43:42.370 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T02:43:42.370 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T02:43:42.570 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:42.616 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:42.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:42 smithi105 ceph-mon[29228]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:43:44.028 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.043+0000 7fbcd192d700 1 -- 172.21.15.184:0/4088250547 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbccc1008b0 msgr2=0x7fbccc100cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:44.028 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.043+0000 7fbcd192d700 1 --2- 172.21.15.184:0/4088250547 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbccc1008b0 0x7fbccc100cd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbcc400b0a0 tx=0x7fbcc400b3b0 comp rx=0 tx=0).stop 2024-04-06T02:43:44.029 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.043+0000 7fbcd192d700 1 -- 172.21.15.184:0/4088250547 shutdown_connections 2024-04-06T02:43:44.029 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.043+0000 7fbcd192d700 1 --2- 172.21.15.184:0/4088250547 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbccc1008b0 0x7fbccc100cd0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:44.029 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.043+0000 7fbcd192d700 1 -- 172.21.15.184:0/4088250547 >> 172.21.15.184:0/4088250547 conn(0x7fbccc0fbe30 msgr2=0x7fbccc0fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:44.029 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.043+0000 7fbcd192d700 1 -- 172.21.15.184:0/4088250547 shutdown_connections 2024-04-06T02:43:44.029 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.043+0000 7fbcd192d700 1 -- 172.21.15.184:0/4088250547 wait complete. 2024-04-06T02:43:44.031 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.044+0000 7fbcd192d700 1 Processor -- start 2024-04-06T02:43:44.031 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.044+0000 7fbcd192d700 1 -- start start 2024-04-06T02:43:44.031 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.044+0000 7fbcd192d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbccc1008b0 0x7fbccc074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:44.031 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.044+0000 7fbcd192d700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc075070 con 0x7fbccc1008b0 2024-04-06T02:43:44.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.044+0000 7fbccaffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbccc1008b0 0x7fbccc074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:44.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.045+0000 7fbccaffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbccc1008b0 0x7fbccc074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.184:48982/0 (socket says 172.21.15.184:48982) 2024-04-06T02:43:44.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.045+0000 7fbccaffd700 1 -- 172.21.15.184:0/1053810121 learned_addr learned my addr 172.21.15.184:0/1053810121 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-04-06T02:43:44.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.045+0000 7fbccaffd700 1 -- 172.21.15.184:0/1053810121 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc4009d00 con 0x7fbccc1008b0 2024-04-06T02:43:44.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.045+0000 7fbccaffd700 1 --2- 172.21.15.184:0/1053810121 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbccc1008b0 0x7fbccc074b30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbcc400b830 tx=0x7fbcc400bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:44.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.045+0000 7fbcd092b700 1 -- 172.21.15.184:0/1053810121 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcc400e040 con 0x7fbccc1008b0 2024-04-06T02:43:44.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.045+0000 7fbcd192d700 1 -- 172.21.15.184:0/1053810121 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbccc073180 con 0x7fbccc1008b0 2024-04-06T02:43:44.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.045+0000 7fbcd092b700 1 -- 172.21.15.184:0/1053810121 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbcc4012af0 con 0x7fbccc1008b0 2024-04-06T02:43:44.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.045+0000 7fbcd092b700 1 -- 172.21.15.184:0/1053810121 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcc401d9f0 con 0x7fbccc1008b0 2024-04-06T02:43:44.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.045+0000 7fbcd192d700 1 -- 172.21.15.184:0/1053810121 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbccc0735a0 con 0x7fbccc1008b0 2024-04-06T02:43:44.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.046+0000 7fbcd192d700 1 -- 172.21.15.184:0/1053810121 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbccc073fb0 con 0x7fbccc1008b0 2024-04-06T02:43:44.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.046+0000 7fbcd092b700 1 -- 172.21.15.184:0/1053810121 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fbcc4012c60 con 0x7fbccc1008b0 2024-04-06T02:43:44.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.046+0000 7fbcd092b700 1 --2- 172.21.15.184:0/1053810121 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fbcb40383c0 0x7fbcb403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:44.033 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.046+0000 7fbcd092b700 1 -- 172.21.15.184:0/1053810121 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbcc404c950 con 0x7fbccc1008b0 2024-04-06T02:43:44.033 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.047+0000 7fbcca7fc700 1 --2- 172.21.15.184:0/1053810121 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fbcb40383c0 0x7fbcb403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:44.033 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.047+0000 7fbcca7fc700 1 --2- 172.21.15.184:0/1053810121 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fbcb40383c0 0x7fbcb403a880 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0006fd0 tx=0x7fbcc0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:44.036 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.052+0000 7fbcd092b700 1 -- 172.21.15.184:0/1053810121 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fbcc401c5c0 con 0x7fbccc1008b0 2024-04-06T02:43:44.298 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.314+0000 7fbcd192d700 1 -- 172.21.15.184:0/1053810121 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbccc18eeb0 con 0x7fbccc1008b0 2024-04-06T02:43:44.299 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.314+0000 7fbcd092b700 1 -- 172.21.15.184:0/1053810121 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fbcc401c3e0 con 0x7fbccc1008b0 2024-04-06T02:43:44.299 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:43:44.300 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"130df578-f3bf-11ee-b649-cb9ed24678a4","modified":"2024-04-06T02:41:29.716000Z","created":"2024-04-06T02:41:29.716000Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T02:43:44.301 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.316+0000 7fbcd192d700 1 -- 172.21.15.184:0/1053810121 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fbcb40383c0 msgr2=0x7fbcb403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:44.301 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.316+0000 7fbcd192d700 1 --2- 172.21.15.184:0/1053810121 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fbcb40383c0 0x7fbcb403a880 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0006fd0 tx=0x7fbcc0006e40 comp rx=0 tx=0).stop 2024-04-06T02:43:44.302 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.316+0000 7fbcd192d700 1 -- 172.21.15.184:0/1053810121 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbccc1008b0 msgr2=0x7fbccc074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:44.302 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.316+0000 7fbcd192d700 1 --2- 172.21.15.184:0/1053810121 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbccc1008b0 0x7fbccc074b30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbcc400b830 tx=0x7fbcc400bd40 comp rx=0 tx=0).stop 2024-04-06T02:43:44.302 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.316+0000 7fbcd192d700 1 -- 172.21.15.184:0/1053810121 shutdown_connections 2024-04-06T02:43:44.302 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.316+0000 7fbcd192d700 1 --2- 172.21.15.184:0/1053810121 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7fbcb40383c0 0x7fbcb403a880 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:44.302 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.316+0000 7fbcd192d700 1 --2- 172.21.15.184:0/1053810121 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbccc1008b0 0x7fbccc074b30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:44.302 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.316+0000 7fbcd192d700 1 -- 172.21.15.184:0/1053810121 >> 172.21.15.184:0/1053810121 conn(0x7fbccc0fbe30 msgr2=0x7fbccc0fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:44.302 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.317+0000 7fbcd192d700 1 -- 172.21.15.184:0/1053810121 shutdown_connections 2024-04-06T02:43:44.302 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:44.317+0000 7fbcd192d700 1 -- 172.21.15.184:0/1053810121 wait complete. 2024-04-06T02:43:44.302 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-04-06T02:43:44.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:44 smithi105 ceph-mon[29228]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:43:44.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:44 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/1053810121' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T02:43:45.874 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T02:43:45.874 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T02:43:46.066 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:46.111 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:46.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:46 smithi105 ceph-mon[29228]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:43:47.451 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.465+0000 7f1a6fe11700 1 -- 172.21.15.184:0/1764126540 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a680ff7a0 msgr2=0x7f1a680ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:47.451 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.465+0000 7f1a6fe11700 1 --2- 172.21.15.184:0/1764126540 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a680ff7a0 0x7f1a680ffbc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1a64009910 tx=0x7f1a64009c20 comp rx=0 tx=0).stop 2024-04-06T02:43:47.451 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.465+0000 7f1a6fe11700 1 -- 172.21.15.184:0/1764126540 shutdown_connections 2024-04-06T02:43:47.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.465+0000 7f1a6fe11700 1 --2- 172.21.15.184:0/1764126540 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a680ff7a0 0x7f1a680ffbc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:47.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.465+0000 7f1a6fe11700 1 -- 172.21.15.184:0/1764126540 >> 172.21.15.184:0/1764126540 conn(0x7f1a680fad20 msgr2=0x7f1a680fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:47.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.465+0000 7f1a6fe11700 1 -- 172.21.15.184:0/1764126540 shutdown_connections 2024-04-06T02:43:47.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.465+0000 7f1a6fe11700 1 -- 172.21.15.184:0/1764126540 wait complete. 2024-04-06T02:43:47.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.466+0000 7f1a6fe11700 1 Processor -- start 2024-04-06T02:43:47.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.466+0000 7f1a6fe11700 1 -- start start 2024-04-06T02:43:47.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.466+0000 7f1a6fe11700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a680ff7a0 0x7f1a68069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:47.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.466+0000 7f1a6fe11700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a681986a0 con 0x7f1a680ff7a0 2024-04-06T02:43:47.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.466+0000 7f1a6dbad700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a680ff7a0 0x7f1a68069430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:47.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.466+0000 7f1a6dbad700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a680ff7a0 0x7f1a68069430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.184:49000/0 (socket says 172.21.15.184:49000) 2024-04-06T02:43:47.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.466+0000 7f1a6dbad700 1 -- 172.21.15.184:0/152545775 learned_addr learned my addr 172.21.15.184:0/152545775 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-04-06T02:43:47.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.467+0000 7f1a6dbad700 1 -- 172.21.15.184:0/152545775 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a640095f0 con 0x7f1a680ff7a0 2024-04-06T02:43:47.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.467+0000 7f1a6dbad700 1 --2- 172.21.15.184:0/152545775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a680ff7a0 0x7f1a68069430 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1a64005b40 tx=0x7f1a64004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:47.453 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.467+0000 7f1a5effd700 1 -- 172.21.15.184:0/152545775 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a6401d070 con 0x7f1a680ff7a0 2024-04-06T02:43:47.454 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.467+0000 7f1a6fe11700 1 -- 172.21.15.184:0/152545775 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a681987e0 con 0x7f1a680ff7a0 2024-04-06T02:43:47.454 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.467+0000 7f1a5effd700 1 -- 172.21.15.184:0/152545775 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1a6400be10 con 0x7f1a680ff7a0 2024-04-06T02:43:47.454 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.467+0000 7f1a5effd700 1 -- 172.21.15.184:0/152545775 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a6400f460 con 0x7f1a680ff7a0 2024-04-06T02:43:47.454 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.467+0000 7f1a6fe11700 1 -- 172.21.15.184:0/152545775 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a68198c80 con 0x7f1a680ff7a0 2024-04-06T02:43:47.454 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.468+0000 7f1a6fe11700 1 -- 172.21.15.184:0/152545775 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a6818e3c0 con 0x7f1a680ff7a0 2024-04-06T02:43:47.454 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.468+0000 7f1a5effd700 1 -- 172.21.15.184:0/152545775 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f1a6400f6d0 con 0x7f1a680ff7a0 2024-04-06T02:43:47.454 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.468+0000 7f1a5effd700 1 --2- 172.21.15.184:0/152545775 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f1a540383c0 0x7f1a5403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:47.454 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.468+0000 7f1a5effd700 1 -- 172.21.15.184:0/152545775 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1a6404c470 con 0x7f1a680ff7a0 2024-04-06T02:43:47.454 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.469+0000 7f1a6d3ac700 1 --2- 172.21.15.184:0/152545775 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f1a540383c0 0x7f1a5403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:47.454 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.469+0000 7f1a6d3ac700 1 --2- 172.21.15.184:0/152545775 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f1a540383c0 0x7f1a5403a880 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1a58006fd0 tx=0x7f1a58006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:47.458 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.473+0000 7f1a5effd700 1 -- 172.21.15.184:0/152545775 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1a6402ba00 con 0x7f1a680ff7a0 2024-04-06T02:43:47.720 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.735+0000 7f1a6fe11700 1 -- 172.21.15.184:0/152545775 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1a6802cc70 con 0x7f1a680ff7a0 2024-04-06T02:43:47.721 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.736+0000 7f1a5effd700 1 -- 172.21.15.184:0/152545775 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f1a64027020 con 0x7f1a680ff7a0 2024-04-06T02:43:47.721 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:43:47.721 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"130df578-f3bf-11ee-b649-cb9ed24678a4","modified":"2024-04-06T02:41:29.716000Z","created":"2024-04-06T02:41:29.716000Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T02:43:47.723 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.738+0000 7f1a6fe11700 1 -- 172.21.15.184:0/152545775 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f1a540383c0 msgr2=0x7f1a5403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:47.723 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.738+0000 7f1a6fe11700 1 --2- 172.21.15.184:0/152545775 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f1a540383c0 0x7f1a5403a880 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1a58006fd0 tx=0x7f1a58006e40 comp rx=0 tx=0).stop 2024-04-06T02:43:47.723 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.738+0000 7f1a6fe11700 1 -- 172.21.15.184:0/152545775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a680ff7a0 msgr2=0x7f1a68069430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:47.723 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.738+0000 7f1a6fe11700 1 --2- 172.21.15.184:0/152545775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a680ff7a0 0x7f1a68069430 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1a64005b40 tx=0x7f1a64004740 comp rx=0 tx=0).stop 2024-04-06T02:43:47.723 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.738+0000 7f1a6fe11700 1 -- 172.21.15.184:0/152545775 shutdown_connections 2024-04-06T02:43:47.724 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.738+0000 7f1a6fe11700 1 --2- 172.21.15.184:0/152545775 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f1a540383c0 0x7f1a5403a880 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:47.724 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.738+0000 7f1a6fe11700 1 --2- 172.21.15.184:0/152545775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a680ff7a0 0x7f1a68069430 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:47.724 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.738+0000 7f1a6fe11700 1 -- 172.21.15.184:0/152545775 >> 172.21.15.184:0/152545775 conn(0x7f1a680fad20 msgr2=0x7f1a680fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:47.724 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.738+0000 7f1a6fe11700 1 -- 172.21.15.184:0/152545775 shutdown_connections 2024-04-06T02:43:47.724 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:47.738+0000 7f1a6fe11700 1 -- 172.21.15.184:0/152545775 wait complete. 2024-04-06T02:43:47.724 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-04-06T02:43:48.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:47 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:48.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:47 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:48.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:47 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:48.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:47 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-06T02:43:48.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:47 smithi105 ceph-mon[29228]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:43:48.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:47 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/152545775' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T02:43:49.094 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:48 smithi105 ceph-mon[29228]: from='mgr.14162 172.21.15.105:0/3866015890' entity='mgr.smithi105.vqqcpx' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-06T02:43:49.094 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:48 smithi105 ceph-mon[29228]: mgrmap e14: smithi105.vqqcpx(active, since 72s) 2024-04-06T02:43:49.268 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T02:43:49.269 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T02:43:49.470 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:49.516 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:50.911 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.926+0000 7f416d4b0700 1 -- 172.21.15.184:0/1332561340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41680ff7a0 msgr2=0x7f41680ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:50.911 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.926+0000 7f416d4b0700 1 --2- 172.21.15.184:0/1332561340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41680ff7a0 0x7f41680ffbc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4158009910 tx=0x7f4158009c20 comp rx=0 tx=0).stop 2024-04-06T02:43:50.911 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.926+0000 7f416d4b0700 1 -- 172.21.15.184:0/1332561340 shutdown_connections 2024-04-06T02:43:50.911 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.926+0000 7f416d4b0700 1 --2- 172.21.15.184:0/1332561340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41680ff7a0 0x7f41680ffbc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:50.912 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.926+0000 7f416d4b0700 1 -- 172.21.15.184:0/1332561340 >> 172.21.15.184:0/1332561340 conn(0x7f41680fad20 msgr2=0x7f41680fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:50.912 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.926+0000 7f416d4b0700 1 -- 172.21.15.184:0/1332561340 shutdown_connections 2024-04-06T02:43:50.912 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.926+0000 7f416d4b0700 1 -- 172.21.15.184:0/1332561340 wait complete. 2024-04-06T02:43:50.912 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.926+0000 7f416d4b0700 1 Processor -- start 2024-04-06T02:43:50.912 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.927+0000 7f416d4b0700 1 -- start start 2024-04-06T02:43:50.912 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.927+0000 7f416d4b0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41680ff7a0 0x7f4168197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:50.912 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.927+0000 7f416d4b0700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41681980b0 con 0x7f41680ff7a0 2024-04-06T02:43:50.914 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.927+0000 7f4166ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41680ff7a0 0x7f4168197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:50.914 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.927+0000 7f4166ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41680ff7a0 0x7f4168197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.184:53344/0 (socket says 172.21.15.184:53344) 2024-04-06T02:43:50.914 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.927+0000 7f4166ffd700 1 -- 172.21.15.184:0/2885545171 learned_addr learned my addr 172.21.15.184:0/2885545171 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-04-06T02:43:50.914 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.927+0000 7f4166ffd700 1 -- 172.21.15.184:0/2885545171 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41580095f0 con 0x7f41680ff7a0 2024-04-06T02:43:50.914 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.927+0000 7f4166ffd700 1 --2- 172.21.15.184:0/2885545171 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41680ff7a0 0x7f4168197b70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4158005b40 tx=0x7f4158004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:50.914 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.927+0000 7f414ffff700 1 -- 172.21.15.184:0/2885545171 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f415801d070 con 0x7f41680ff7a0 2024-04-06T02:43:50.914 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.928+0000 7f416d4b0700 1 -- 172.21.15.184:0/2885545171 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41681982b0 con 0x7f41680ff7a0 2024-04-06T02:43:50.914 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.928+0000 7f414ffff700 1 -- 172.21.15.184:0/2885545171 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f415800be10 con 0x7f41680ff7a0 2024-04-06T02:43:50.914 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.928+0000 7f414ffff700 1 -- 172.21.15.184:0/2885545171 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f415800f460 con 0x7f41680ff7a0 2024-04-06T02:43:50.914 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.928+0000 7f416d4b0700 1 -- 172.21.15.184:0/2885545171 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4168198750 con 0x7f41680ff7a0 2024-04-06T02:43:50.914 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.928+0000 7f414ffff700 1 -- 172.21.15.184:0/2885545171 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 14) v1 ==== 45354+0+0 (secure 0 0 0) 0x7f415800b960 con 0x7f41680ff7a0 2024-04-06T02:43:50.914 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.928+0000 7f416d4b0700 1 -- 172.21.15.184:0/2885545171 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f416818d2b0 con 0x7f41680ff7a0 2024-04-06T02:43:50.915 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.929+0000 7f414ffff700 1 --2- 172.21.15.184:0/2885545171 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f4150038410 0x7f415003a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:50.915 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.929+0000 7f414ffff700 1 -- 172.21.15.184:0/2885545171 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f415804c490 con 0x7f41680ff7a0 2024-04-06T02:43:50.915 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.929+0000 7f41667fc700 1 -- 172.21.15.184:0/2885545171 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f4150038410 msgr2=0x7f415003a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/1949998385 2024-04-06T02:43:50.915 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.929+0000 7f41667fc700 1 --2- 172.21.15.184:0/2885545171 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f4150038410 0x7f415003a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T02:43:50.918 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:50.933+0000 7f414ffff700 1 -- 172.21.15.184:0/2885545171 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f415802a360 con 0x7f41680ff7a0 2024-04-06T02:43:51.114 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:51.130+0000 7f41667fc700 1 -- 172.21.15.184:0/2885545171 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f4150038410 msgr2=0x7f415003a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/1949998385 2024-04-06T02:43:51.115 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:51.130+0000 7f41667fc700 1 --2- 172.21.15.184:0/2885545171 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f4150038410 0x7f415003a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T02:43:51.180 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:51.196+0000 7f416d4b0700 1 -- 172.21.15.184:0/2885545171 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f416802cc70 con 0x7f41680ff7a0 2024-04-06T02:43:51.181 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:51.196+0000 7f414ffff700 1 -- 172.21.15.184:0/2885545171 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f4158027070 con 0x7f41680ff7a0 2024-04-06T02:43:51.181 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:43:51.181 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"130df578-f3bf-11ee-b649-cb9ed24678a4","modified":"2024-04-06T02:41:29.716000Z","created":"2024-04-06T02:41:29.716000Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T02:43:51.183 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:51.198+0000 7f416d4b0700 1 -- 172.21.15.184:0/2885545171 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f4150038410 msgr2=0x7f415003a8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T02:43:51.183 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:51.198+0000 7f416d4b0700 1 --2- 172.21.15.184:0/2885545171 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f4150038410 0x7f415003a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:51.183 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:51.198+0000 7f416d4b0700 1 -- 172.21.15.184:0/2885545171 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41680ff7a0 msgr2=0x7f4168197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:51.183 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:51.198+0000 7f416d4b0700 1 --2- 172.21.15.184:0/2885545171 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41680ff7a0 0x7f4168197b70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4158005b40 tx=0x7f4158004740 comp rx=0 tx=0).stop 2024-04-06T02:43:51.183 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:51.198+0000 7f416d4b0700 1 -- 172.21.15.184:0/2885545171 shutdown_connections 2024-04-06T02:43:51.184 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:51.198+0000 7f416d4b0700 1 --2- 172.21.15.184:0/2885545171 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f4150038410 0x7f415003a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:51.184 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:51.198+0000 7f416d4b0700 1 --2- 172.21.15.184:0/2885545171 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41680ff7a0 0x7f4168197b70 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:51.184 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:51.198+0000 7f416d4b0700 1 -- 172.21.15.184:0/2885545171 >> 172.21.15.184:0/2885545171 conn(0x7f41680fad20 msgr2=0x7f41680fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:51.184 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:51.199+0000 7f416d4b0700 1 -- 172.21.15.184:0/2885545171 shutdown_connections 2024-04-06T02:43:51.184 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:51.199+0000 7f416d4b0700 1 -- 172.21.15.184:0/2885545171 wait complete. 2024-04-06T02:43:51.184 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-04-06T02:43:51.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:51 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/2885545171' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T02:43:52.746 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T02:43:52.747 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T02:43:52.944 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:52.990 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:54.282 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.297+0000 7f3b5c919700 1 -- 172.21.15.184:0/3723336317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b580ff7a0 msgr2=0x7f3b580ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:54.282 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.297+0000 7f3b5c919700 1 --2- 172.21.15.184:0/3723336317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b580ff7a0 0x7f3b580ffbc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3b48009910 tx=0x7f3b48009c20 comp rx=0 tx=0).stop 2024-04-06T02:43:54.282 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.297+0000 7f3b5c919700 1 -- 172.21.15.184:0/3723336317 shutdown_connections 2024-04-06T02:43:54.282 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.297+0000 7f3b5c919700 1 --2- 172.21.15.184:0/3723336317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b580ff7a0 0x7f3b580ffbc0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:54.282 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.297+0000 7f3b5c919700 1 -- 172.21.15.184:0/3723336317 >> 172.21.15.184:0/3723336317 conn(0x7f3b580fad20 msgr2=0x7f3b580fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:54.282 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.297+0000 7f3b5c919700 1 -- 172.21.15.184:0/3723336317 shutdown_connections 2024-04-06T02:43:54.282 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.297+0000 7f3b5c919700 1 -- 172.21.15.184:0/3723336317 wait complete. 2024-04-06T02:43:54.284 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.297+0000 7f3b5c919700 1 Processor -- start 2024-04-06T02:43:54.284 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.298+0000 7f3b5c919700 1 -- start start 2024-04-06T02:43:54.284 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.298+0000 7f3b5c919700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b580ff7a0 0x7f3b581959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:54.284 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.298+0000 7f3b5c919700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b58195f00 con 0x7f3b580ff7a0 2024-04-06T02:43:54.284 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.298+0000 7f3b5659c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b580ff7a0 0x7f3b581959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:54.284 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.298+0000 7f3b5659c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b580ff7a0 0x7f3b581959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.184:53362/0 (socket says 172.21.15.184:53362) 2024-04-06T02:43:54.284 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.298+0000 7f3b5659c700 1 -- 172.21.15.184:0/339337027 learned_addr learned my addr 172.21.15.184:0/339337027 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-04-06T02:43:54.284 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.298+0000 7f3b5659c700 1 -- 172.21.15.184:0/339337027 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b480095f0 con 0x7f3b580ff7a0 2024-04-06T02:43:54.284 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.299+0000 7f3b5659c700 1 --2- 172.21.15.184:0/339337027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b580ff7a0 0x7f3b581959c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3b48005b40 tx=0x7f3b48004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:54.284 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.299+0000 7f3b4ffff700 1 -- 172.21.15.184:0/339337027 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b4801d070 con 0x7f3b580ff7a0 2024-04-06T02:43:54.284 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.299+0000 7f3b5c919700 1 -- 172.21.15.184:0/339337027 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b58196100 con 0x7f3b580ff7a0 2024-04-06T02:43:54.285 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.299+0000 7f3b4ffff700 1 -- 172.21.15.184:0/339337027 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3b4800be10 con 0x7f3b580ff7a0 2024-04-06T02:43:54.285 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.299+0000 7f3b4ffff700 1 -- 172.21.15.184:0/339337027 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b4800f460 con 0x7f3b580ff7a0 2024-04-06T02:43:54.285 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.299+0000 7f3b5c919700 1 -- 172.21.15.184:0/339337027 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b581965a0 con 0x7f3b580ff7a0 2024-04-06T02:43:54.285 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.300+0000 7f3b4ffff700 1 -- 172.21.15.184:0/339337027 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 14) v1 ==== 45354+0+0 (secure 0 0 0) 0x7f3b4800b960 con 0x7f3b580ff7a0 2024-04-06T02:43:54.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.300+0000 7f3b5c919700 1 -- 172.21.15.184:0/339337027 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b5818f3d0 con 0x7f3b580ff7a0 2024-04-06T02:43:54.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.300+0000 7f3b4ffff700 1 --2- 172.21.15.184:0/339337027 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f3b44038410 0x7f3b4403a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:54.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.300+0000 7f3b4ffff700 1 -- 172.21.15.184:0/339337027 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3b4804c4f0 con 0x7f3b580ff7a0 2024-04-06T02:43:54.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.300+0000 7f3b4dd9b700 1 -- 172.21.15.184:0/339337027 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f3b44038410 msgr2=0x7f3b4403a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/1949998385 2024-04-06T02:43:54.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.300+0000 7f3b4dd9b700 1 --2- 172.21.15.184:0/339337027 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f3b44038410 0x7f3b4403a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T02:43:54.289 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.305+0000 7f3b4ffff700 1 -- 172.21.15.184:0/339337027 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3b4802a360 con 0x7f3b580ff7a0 2024-04-06T02:43:54.485 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.501+0000 7f3b4dd9b700 1 -- 172.21.15.184:0/339337027 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f3b44038410 msgr2=0x7f3b4403a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/1949998385 2024-04-06T02:43:54.486 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.501+0000 7f3b4dd9b700 1 --2- 172.21.15.184:0/339337027 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f3b44038410 0x7f3b4403a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-06T02:43:54.553 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.569+0000 7f3b5c919700 1 -- 172.21.15.184:0/339337027 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3b5802cc70 con 0x7f3b580ff7a0 2024-04-06T02:43:54.554 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.570+0000 7f3b4ffff700 1 -- 172.21.15.184:0/339337027 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f3b48027030 con 0x7f3b580ff7a0 2024-04-06T02:43:54.555 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:43:54.555 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"130df578-f3bf-11ee-b649-cb9ed24678a4","modified":"2024-04-06T02:41:29.716000Z","created":"2024-04-06T02:41:29.716000Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T02:43:54.557 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.572+0000 7f3b5c919700 1 -- 172.21.15.184:0/339337027 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f3b44038410 msgr2=0x7f3b4403a8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T02:43:54.557 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.572+0000 7f3b5c919700 1 --2- 172.21.15.184:0/339337027 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f3b44038410 0x7f3b4403a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:54.557 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.572+0000 7f3b5c919700 1 -- 172.21.15.184:0/339337027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b580ff7a0 msgr2=0x7f3b581959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:54.557 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.572+0000 7f3b5c919700 1 --2- 172.21.15.184:0/339337027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b580ff7a0 0x7f3b581959c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3b48005b40 tx=0x7f3b48004740 comp rx=0 tx=0).stop 2024-04-06T02:43:54.557 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.572+0000 7f3b5c919700 1 -- 172.21.15.184:0/339337027 shutdown_connections 2024-04-06T02:43:54.557 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.572+0000 7f3b5c919700 1 --2- 172.21.15.184:0/339337027 >> [v2:172.21.15.105:6800/1949998385,v1:172.21.15.105:6801/1949998385] conn(0x7f3b44038410 0x7f3b4403a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:54.557 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.572+0000 7f3b5c919700 1 --2- 172.21.15.184:0/339337027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b580ff7a0 0x7f3b581959c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:54.557 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.572+0000 7f3b5c919700 1 -- 172.21.15.184:0/339337027 >> 172.21.15.184:0/339337027 conn(0x7f3b580fad20 msgr2=0x7f3b58069100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:54.557 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.572+0000 7f3b5c919700 1 -- 172.21.15.184:0/339337027 shutdown_connections 2024-04-06T02:43:54.557 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:54.572+0000 7f3b5c919700 1 -- 172.21.15.184:0/339337027 wait complete. 2024-04-06T02:43:54.558 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-04-06T02:43:54.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:54 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/339337027' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T02:43:54.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:54 smithi105 ceph-mon[29228]: Active manager daemon smithi105.vqqcpx restarted 2024-04-06T02:43:54.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:54 smithi105 ceph-mon[29228]: Activating manager daemon smithi105.vqqcpx 2024-04-06T02:43:54.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:54 smithi105 ceph-mon[29228]: osdmap e5: 0 total, 0 up, 0 in 2024-04-06T02:43:54.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:54 smithi105 ceph-mon[29228]: mgrmap e15: smithi105.vqqcpx(active, starting, since 0.00312578s) 2024-04-06T02:43:54.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:54 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-04-06T02:43:54.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:54 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mgr metadata", "who": "smithi105.vqqcpx", "id": "smithi105.vqqcpx"}]: dispatch 2024-04-06T02:43:54.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:54 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T02:43:54.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:54 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T02:43:54.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:54 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T02:43:54.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:54 smithi105 ceph-mon[29228]: Manager daemon smithi105.vqqcpx is now available 2024-04-06T02:43:54.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:54 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:43:54.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:54 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:43:55.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:55 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:55.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:55 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:43:55.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:55 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi105.vqqcpx/mirror_snapshot_schedule"}]: dispatch 2024-04-06T02:43:55.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:55 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi105.vqqcpx/trash_purge_schedule"}]: dispatch 2024-04-06T02:43:55.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:55 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:55.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:55 smithi105 ceph-mon[29228]: mgrmap e16: smithi105.vqqcpx(active, since 1.00551s) 2024-04-06T02:43:55.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:55 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:56.121 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T02:43:56.121 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T02:43:56.318 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:56.364 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-06T02:43:56.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:56 smithi105 ceph-mon[29228]: [06/Apr/2024:02:43:55] ENGINE Bus STARTING 2024-04-06T02:43:56.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:56 smithi105 ceph-mon[29228]: [06/Apr/2024:02:43:55] ENGINE Serving on https://172.21.15.105:7150 2024-04-06T02:43:56.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:56 smithi105 ceph-mon[29228]: [06/Apr/2024:02:43:55] ENGINE Serving on http://172.21.15.105:8765 2024-04-06T02:43:56.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:56 smithi105 ceph-mon[29228]: [06/Apr/2024:02:43:55] ENGINE Bus STARTED 2024-04-06T02:43:56.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:56 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:58.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.408+0000 7f64a8d12700 1 -- 172.21.15.184:0/52401435 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64a40ff790 msgr2=0x7f64a40ffbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:58.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.408+0000 7f64a8d12700 1 --2- 172.21.15.184:0/52401435 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64a40ff790 0x7f64a40ffbb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6498009910 tx=0x7f6498009c20 comp rx=0 tx=0).stop 2024-04-06T02:43:58.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.408+0000 7f64a8d12700 1 -- 172.21.15.184:0/52401435 shutdown_connections 2024-04-06T02:43:58.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.408+0000 7f64a8d12700 1 --2- 172.21.15.184:0/52401435 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64a40ff790 0x7f64a40ffbb0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:58.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.408+0000 7f64a8d12700 1 -- 172.21.15.184:0/52401435 >> 172.21.15.184:0/52401435 conn(0x7f64a40fad30 msgr2=0x7f64a40fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:58.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.408+0000 7f64a8d12700 1 -- 172.21.15.184:0/52401435 shutdown_connections 2024-04-06T02:43:58.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.408+0000 7f64a8d12700 1 -- 172.21.15.184:0/52401435 wait complete. 2024-04-06T02:43:58.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.409+0000 7f64a8d12700 1 Processor -- start 2024-04-06T02:43:58.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.409+0000 7f64a8d12700 1 -- start start 2024-04-06T02:43:58.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.409+0000 7f64a8d12700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64a40ff790 0x7f64a4069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:58.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.409+0000 7f64a8d12700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a41986c0 con 0x7f64a40ff790 2024-04-06T02:43:58.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.409+0000 7f64a259c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64a40ff790 0x7f64a4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:58.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.409+0000 7f64a259c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64a40ff790 0x7f64a4069410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.184:53390/0 (socket says 172.21.15.184:53390) 2024-04-06T02:43:58.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.409+0000 7f64a259c700 1 -- 172.21.15.184:0/3736704694 learned_addr learned my addr 172.21.15.184:0/3736704694 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-04-06T02:43:58.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.410+0000 7f64a259c700 1 -- 172.21.15.184:0/3736704694 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64980095f0 con 0x7f64a40ff790 2024-04-06T02:43:58.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.410+0000 7f64a259c700 1 --2- 172.21.15.184:0/3736704694 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64a40ff790 0x7f64a4069410 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6498005b40 tx=0x7f6498004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:58.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.410+0000 7f64937fe700 1 -- 172.21.15.184:0/3736704694 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f649801d070 con 0x7f64a40ff790 2024-04-06T02:43:58.397 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.410+0000 7f64a8d12700 1 -- 172.21.15.184:0/3736704694 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a4198800 con 0x7f64a40ff790 2024-04-06T02:43:58.397 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.410+0000 7f64937fe700 1 -- 172.21.15.184:0/3736704694 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f649800be10 con 0x7f64a40ff790 2024-04-06T02:43:58.397 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.410+0000 7f64a8d12700 1 -- 172.21.15.184:0/3736704694 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a4198ca0 con 0x7f64a40ff790 2024-04-06T02:43:58.397 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.410+0000 7f64937fe700 1 -- 172.21.15.184:0/3736704694 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f649800f460 con 0x7f64a40ff790 2024-04-06T02:43:58.398 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.411+0000 7f64937fe700 1 -- 172.21.15.184:0/3736704694 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7f649800f5c0 con 0x7f64a40ff790 2024-04-06T02:43:58.398 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.411+0000 7f64a8d12700 1 -- 172.21.15.184:0/3736704694 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a418e3b0 con 0x7f64a40ff790 2024-04-06T02:43:58.398 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.411+0000 7f64937fe700 1 --2- 172.21.15.184:0/3736704694 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f648c0384b0 0x7f648c03a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:43:58.398 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.411+0000 7f64937fe700 1 -- 172.21.15.184:0/3736704694 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f649804c6c0 con 0x7f64a40ff790 2024-04-06T02:43:58.398 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.411+0000 7f64a1d9b700 1 --2- 172.21.15.184:0/3736704694 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f648c0384b0 0x7f648c03a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:43:58.398 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.412+0000 7f64a1d9b700 1 --2- 172.21.15.184:0/3736704694 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f648c0384b0 0x7f648c03a970 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6494006fd0 tx=0x7f6494006e40 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:43:58.401 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.417+0000 7f64937fe700 1 -- 172.21.15.184:0/3736704694 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f649802abe0 con 0x7f64a40ff790 2024-04-06T02:43:58.670 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.685+0000 7f64a8d12700 1 -- 172.21.15.184:0/3736704694 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64a402cc70 con 0x7f64a40ff790 2024-04-06T02:43:58.671 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.686+0000 7f64937fe700 1 -- 172.21.15.184:0/3736704694 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f6498027030 con 0x7f64a40ff790 2024-04-06T02:43:58.671 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:43:58.671 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"130df578-f3bf-11ee-b649-cb9ed24678a4","modified":"2024-04-06T02:41:29.716000Z","created":"2024-04-06T02:41:29.716000Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T02:43:58.673 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.688+0000 7f64a8d12700 1 -- 172.21.15.184:0/3736704694 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f648c0384b0 msgr2=0x7f648c03a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:58.673 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.688+0000 7f64a8d12700 1 --2- 172.21.15.184:0/3736704694 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f648c0384b0 0x7f648c03a970 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6494006fd0 tx=0x7f6494006e40 comp rx=0 tx=0).stop 2024-04-06T02:43:58.673 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.688+0000 7f64a8d12700 1 -- 172.21.15.184:0/3736704694 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64a40ff790 msgr2=0x7f64a4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:43:58.673 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.688+0000 7f64a8d12700 1 --2- 172.21.15.184:0/3736704694 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64a40ff790 0x7f64a4069410 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6498005b40 tx=0x7f6498004740 comp rx=0 tx=0).stop 2024-04-06T02:43:58.673 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.688+0000 7f64a8d12700 1 -- 172.21.15.184:0/3736704694 shutdown_connections 2024-04-06T02:43:58.673 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.688+0000 7f64a8d12700 1 --2- 172.21.15.184:0/3736704694 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f648c0384b0 0x7f648c03a970 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:58.674 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.688+0000 7f64a8d12700 1 --2- 172.21.15.184:0/3736704694 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64a40ff790 0x7f64a4069410 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:43:58.674 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.688+0000 7f64a8d12700 1 -- 172.21.15.184:0/3736704694 >> 172.21.15.184:0/3736704694 conn(0x7f64a40fad30 msgr2=0x7f64a40fb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:43:58.674 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.688+0000 7f64a8d12700 1 -- 172.21.15.184:0/3736704694 shutdown_connections 2024-04-06T02:43:58.674 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:43:58.688+0000 7f64a8d12700 1 -- 172.21.15.184:0/3736704694 wait complete. 2024-04-06T02:43:58.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:58 smithi105 ceph-mon[29228]: mgrmap e17: smithi105.vqqcpx(active, since 2s) 2024-04-06T02:43:58.674 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-04-06T02:43:59.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:59 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/3736704694' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T02:43:59.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:59 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:43:59.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:43:59 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:00.882 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T02:44:00.882 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T02:44:00.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:00 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:00.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:00 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:00.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:00 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config rm", "who": "osd/host:smithi105", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:44:00.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:00 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:00.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:00 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:00.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:00 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:00.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:00 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:00.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:00 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:44:00.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:00 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:44:00.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:00 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:44:01.079 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:44:01.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:01 smithi105 ceph-mon[29228]: Updating smithi105:/etc/ceph/ceph.conf 2024-04-06T02:44:01.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:01 smithi105 ceph-mon[29228]: Updating smithi184:/etc/ceph/ceph.conf 2024-04-06T02:44:01.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:01 smithi105 ceph-mon[29228]: Updating smithi184:/var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:44:01.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:01 smithi105 ceph-mon[29228]: Updating smithi105:/var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:44:01.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:01 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:01.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:01 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:01.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:01 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:01.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:01 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:01.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:01 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:01.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:01 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi184", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-06T02:44:01.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:01 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi184", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-06T02:44:01.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:01 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:44:02.627 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.642+0000 7f66d6882700 1 -- 172.21.15.184:0/2793673028 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66d0105730 msgr2=0x7f66d0105b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:44:02.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.642+0000 7f66d6882700 1 --2- 172.21.15.184:0/2793673028 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66d0105730 0x7f66d0105b10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f66c0009910 tx=0x7f66c0009c20 comp rx=0 tx=0).stop 2024-04-06T02:44:02.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.642+0000 7f66d6882700 1 -- 172.21.15.184:0/2793673028 shutdown_connections 2024-04-06T02:44:02.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.642+0000 7f66d6882700 1 --2- 172.21.15.184:0/2793673028 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66d0105730 0x7f66d0105b10 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:02.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.642+0000 7f66d6882700 1 -- 172.21.15.184:0/2793673028 >> 172.21.15.184:0/2793673028 conn(0x7f66d00fb240 msgr2=0x7f66d00fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:44:02.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.642+0000 7f66d6882700 1 -- 172.21.15.184:0/2793673028 shutdown_connections 2024-04-06T02:44:02.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.642+0000 7f66d6882700 1 -- 172.21.15.184:0/2793673028 wait complete. 2024-04-06T02:44:02.630 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.643+0000 7f66d6882700 1 Processor -- start 2024-04-06T02:44:02.630 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.643+0000 7f66d6882700 1 -- start start 2024-04-06T02:44:02.630 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.643+0000 7f66d6882700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66d0105730 0x7f66d0195d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:44:02.630 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.643+0000 7f66d6882700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d0196250 con 0x7f66d0105730 2024-04-06T02:44:02.631 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.643+0000 7f66cffff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66d0105730 0x7f66d0195d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:44:02.631 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.643+0000 7f66cffff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66d0105730 0x7f66d0195d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.184:51730/0 (socket says 172.21.15.184:51730) 2024-04-06T02:44:02.631 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.643+0000 7f66cffff700 1 -- 172.21.15.184:0/1636700283 learned_addr learned my addr 172.21.15.184:0/1636700283 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-04-06T02:44:02.631 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.644+0000 7f66cffff700 1 -- 172.21.15.184:0/1636700283 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c00095f0 con 0x7f66d0105730 2024-04-06T02:44:02.631 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.644+0000 7f66cffff700 1 --2- 172.21.15.184:0/1636700283 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66d0105730 0x7f66d0195d10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f66c0005b40 tx=0x7f66c0004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:44:02.631 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.644+0000 7f66cdffb700 1 -- 172.21.15.184:0/1636700283 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66c001d070 con 0x7f66d0105730 2024-04-06T02:44:02.631 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.644+0000 7f66d6882700 1 -- 172.21.15.184:0/1636700283 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d0196450 con 0x7f66d0105730 2024-04-06T02:44:02.631 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.644+0000 7f66cdffb700 1 -- 172.21.15.184:0/1636700283 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f66c00044d0 con 0x7f66d0105730 2024-04-06T02:44:02.631 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.644+0000 7f66cdffb700 1 -- 172.21.15.184:0/1636700283 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66c0003910 con 0x7f66d0105730 2024-04-06T02:44:02.631 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.644+0000 7f66d6882700 1 -- 172.21.15.184:0/1636700283 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d01968d0 con 0x7f66d0105730 2024-04-06T02:44:02.631 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.645+0000 7f66d6882700 1 -- 172.21.15.184:0/1636700283 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d00623c0 con 0x7f66d0105730 2024-04-06T02:44:02.631 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.645+0000 7f66cdffb700 1 -- 172.21.15.184:0/1636700283 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7f66c0003b90 con 0x7f66d0105730 2024-04-06T02:44:02.631 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.645+0000 7f66cdffb700 1 --2- 172.21.15.184:0/1636700283 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f66bc0384b0 0x7f66bc03a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:44:02.632 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.645+0000 7f66cdffb700 1 -- 172.21.15.184:0/1636700283 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f66c004c4c0 con 0x7f66d0105730 2024-04-06T02:44:02.632 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.645+0000 7f66c7fff700 1 --2- 172.21.15.184:0/1636700283 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f66bc0384b0 0x7f66bc03a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:44:02.632 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.646+0000 7f66c7fff700 1 --2- 172.21.15.184:0/1636700283 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f66bc0384b0 0x7f66bc03a970 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f66b8006fd0 tx=0x7f66b8006e40 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:44:02.635 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.650+0000 7f66cdffb700 1 -- 172.21.15.184:0/1636700283 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f66c002a540 con 0x7f66d0105730 2024-04-06T02:44:02.912 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.927+0000 7f66d6882700 1 -- 172.21.15.184:0/1636700283 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66d019a840 con 0x7f66d0105730 2024-04-06T02:44:02.913 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.928+0000 7f66cdffb700 1 -- 172.21.15.184:0/1636700283 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f66c002a360 con 0x7f66d0105730 2024-04-06T02:44:02.913 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:44:02.913 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"130df578-f3bf-11ee-b649-cb9ed24678a4","modified":"2024-04-06T02:41:29.716000Z","created":"2024-04-06T02:41:29.716000Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T02:44:02.915 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.931+0000 7f66d6882700 1 -- 172.21.15.184:0/1636700283 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f66bc0384b0 msgr2=0x7f66bc03a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:44:02.916 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.931+0000 7f66d6882700 1 --2- 172.21.15.184:0/1636700283 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f66bc0384b0 0x7f66bc03a970 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f66b8006fd0 tx=0x7f66b8006e40 comp rx=0 tx=0).stop 2024-04-06T02:44:02.916 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.931+0000 7f66d6882700 1 -- 172.21.15.184:0/1636700283 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66d0105730 msgr2=0x7f66d0195d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:44:02.916 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.931+0000 7f66d6882700 1 --2- 172.21.15.184:0/1636700283 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66d0105730 0x7f66d0195d10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f66c0005b40 tx=0x7f66c0004c80 comp rx=0 tx=0).stop 2024-04-06T02:44:02.916 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.931+0000 7f66d6882700 1 -- 172.21.15.184:0/1636700283 shutdown_connections 2024-04-06T02:44:02.916 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.931+0000 7f66d6882700 1 --2- 172.21.15.184:0/1636700283 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f66bc0384b0 0x7f66bc03a970 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:02.916 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.931+0000 7f66d6882700 1 --2- 172.21.15.184:0/1636700283 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66d0105730 0x7f66d0195d10 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:02.916 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.931+0000 7f66d6882700 1 -- 172.21.15.184:0/1636700283 >> 172.21.15.184:0/1636700283 conn(0x7f66d00fb240 msgr2=0x7f66d0107b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:44:02.916 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.931+0000 7f66d6882700 1 -- 172.21.15.184:0/1636700283 shutdown_connections 2024-04-06T02:44:02.916 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:02.931+0000 7f66d6882700 1 -- 172.21.15.184:0/1636700283 wait complete. 2024-04-06T02:44:02.917 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-04-06T02:44:02.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:02 smithi105 ceph-mon[29228]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:44:02.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:02 smithi105 ceph-mon[29228]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:44:02.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:02 smithi105 ceph-mon[29228]: Updating smithi184:/var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T02:44:02.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:02 smithi105 ceph-mon[29228]: Updating smithi105:/var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T02:44:02.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:02 smithi105 ceph-mon[29228]: Deploying daemon ceph-exporter.smithi184 on smithi184 2024-04-06T02:44:03.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:03 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/1636700283' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T02:44:05.128 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T02:44:05.128 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T02:44:05.328 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:44:06.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:06 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:06.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:06 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:06.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:06 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:06.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:06 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:06.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:06 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi184", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T02:44:06.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:06 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi184", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-06T02:44:06.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:06 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:44:06.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:06 smithi105 ceph-mon[29228]: Deploying daemon crash.smithi184 on smithi184 2024-04-06T02:44:06.877 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.890+0000 7f5818741700 1 -- 172.21.15.184:0/3644463865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5810105780 msgr2=0x7f5810105b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:44:06.877 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.890+0000 7f5818741700 1 --2- 172.21.15.184:0/3644463865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5810105780 0x7f5810105b60 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f580c009910 tx=0x7f580c009c20 comp rx=0 tx=0).stop 2024-04-06T02:44:06.878 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.890+0000 7f5818741700 1 -- 172.21.15.184:0/3644463865 shutdown_connections 2024-04-06T02:44:06.878 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.890+0000 7f5818741700 1 --2- 172.21.15.184:0/3644463865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5810105780 0x7f5810105b60 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:06.878 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.890+0000 7f5818741700 1 -- 172.21.15.184:0/3644463865 >> 172.21.15.184:0/3644463865 conn(0x7f58100752b0 msgr2=0x7f58100756c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:44:06.878 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.890+0000 7f5818741700 1 -- 172.21.15.184:0/3644463865 shutdown_connections 2024-04-06T02:44:06.878 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.890+0000 7f5818741700 1 -- 172.21.15.184:0/3644463865 wait complete. 2024-04-06T02:44:06.878 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.891+0000 7f5818741700 1 Processor -- start 2024-04-06T02:44:06.878 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.891+0000 7f5818741700 1 -- start start 2024-04-06T02:44:06.878 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.891+0000 7f5818741700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5810105780 0x7f581019aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:44:06.878 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.891+0000 7f5818741700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581019b010 con 0x7f5810105780 2024-04-06T02:44:06.878 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.892+0000 7f58164dd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5810105780 0x7f581019aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:44:06.878 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.892+0000 7f58164dd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5810105780 0x7f581019aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.184:51754/0 (socket says 172.21.15.184:51754) 2024-04-06T02:44:06.878 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.892+0000 7f58164dd700 1 -- 172.21.15.184:0/1400682956 learned_addr learned my addr 172.21.15.184:0/1400682956 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-04-06T02:44:06.878 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.892+0000 7f58164dd700 1 -- 172.21.15.184:0/1400682956 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580c0095f0 con 0x7f5810105780 2024-04-06T02:44:06.879 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.892+0000 7f58164dd700 1 --2- 172.21.15.184:0/1400682956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5810105780 0x7f581019aad0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f580c005b40 tx=0x7f580c004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:44:06.879 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.892+0000 7f58077fe700 1 -- 172.21.15.184:0/1400682956 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f580c01d070 con 0x7f5810105780 2024-04-06T02:44:06.879 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.892+0000 7f5818741700 1 -- 172.21.15.184:0/1400682956 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f581019b210 con 0x7f5810105780 2024-04-06T02:44:06.879 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.892+0000 7f58077fe700 1 -- 172.21.15.184:0/1400682956 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f580c0044d0 con 0x7f5810105780 2024-04-06T02:44:06.879 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.892+0000 7f58077fe700 1 -- 172.21.15.184:0/1400682956 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f580c003930 con 0x7f5810105780 2024-04-06T02:44:06.879 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.892+0000 7f5818741700 1 -- 172.21.15.184:0/1400682956 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f581019b5d0 con 0x7f5810105780 2024-04-06T02:44:06.880 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.893+0000 7f5818741700 1 -- 172.21.15.184:0/1400682956 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58100623c0 con 0x7f5810105780 2024-04-06T02:44:06.880 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.893+0000 7f58077fe700 1 -- 172.21.15.184:0/1400682956 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7f580c00b960 con 0x7f5810105780 2024-04-06T02:44:06.880 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.893+0000 7f58077fe700 1 --2- 172.21.15.184:0/1400682956 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f57fc0384b0 0x7f57fc03a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:44:06.880 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.893+0000 7f58077fe700 1 -- 172.21.15.184:0/1400682956 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f580c04c4a0 con 0x7f5810105780 2024-04-06T02:44:06.880 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.894+0000 7f5815cdc700 1 --2- 172.21.15.184:0/1400682956 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f57fc0384b0 0x7f57fc03a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:44:06.880 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.894+0000 7f5815cdc700 1 --2- 172.21.15.184:0/1400682956 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f57fc0384b0 0x7f57fc03a970 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5800006fd0 tx=0x7f5800006e40 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:44:06.883 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:06.899+0000 7f58077fe700 1 -- 172.21.15.184:0/1400682956 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f580c02a540 con 0x7f5810105780 2024-04-06T02:44:07.155 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:07.170+0000 7f5818741700 1 -- 172.21.15.184:0/1400682956 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f58101035f0 con 0x7f5810105780 2024-04-06T02:44:07.156 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:07.171+0000 7f58077fe700 1 -- 172.21.15.184:0/1400682956 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f580c02a360 con 0x7f5810105780 2024-04-06T02:44:07.156 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:44:07.156 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"130df578-f3bf-11ee-b649-cb9ed24678a4","modified":"2024-04-06T02:41:29.716000Z","created":"2024-04-06T02:41:29.716000Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T02:44:07.159 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:07.173+0000 7f5818741700 1 -- 172.21.15.184:0/1400682956 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f57fc0384b0 msgr2=0x7f57fc03a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:44:07.159 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:07.173+0000 7f5818741700 1 --2- 172.21.15.184:0/1400682956 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f57fc0384b0 0x7f57fc03a970 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5800006fd0 tx=0x7f5800006e40 comp rx=0 tx=0).stop 2024-04-06T02:44:07.159 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:07.173+0000 7f5818741700 1 -- 172.21.15.184:0/1400682956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5810105780 msgr2=0x7f581019aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:44:07.159 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:07.173+0000 7f5818741700 1 --2- 172.21.15.184:0/1400682956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5810105780 0x7f581019aad0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f580c005b40 tx=0x7f580c004c80 comp rx=0 tx=0).stop 2024-04-06T02:44:07.160 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:07.173+0000 7f5818741700 1 -- 172.21.15.184:0/1400682956 shutdown_connections 2024-04-06T02:44:07.160 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:07.173+0000 7f5818741700 1 --2- 172.21.15.184:0/1400682956 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f57fc0384b0 0x7f57fc03a970 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:07.160 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:07.173+0000 7f5818741700 1 --2- 172.21.15.184:0/1400682956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5810105780 0x7f581019aad0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:07.160 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:07.173+0000 7f5818741700 1 -- 172.21.15.184:0/1400682956 >> 172.21.15.184:0/1400682956 conn(0x7f58100752b0 msgr2=0x7f58100fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:44:07.160 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:07.173+0000 7f5818741700 1 -- 172.21.15.184:0/1400682956 shutdown_connections 2024-04-06T02:44:07.160 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:07.173+0000 7f5818741700 1 -- 172.21.15.184:0/1400682956 wait complete. 2024-04-06T02:44:07.161 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-04-06T02:44:07.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:07 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/1400682956' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T02:44:08.762 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T02:44:08.762 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T02:44:09.238 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:44:10.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:10 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:10.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:10 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:10.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:10 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:10.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:10 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:10.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:10 smithi105 ceph-mon[29228]: Deploying daemon node-exporter.smithi184 on smithi184 2024-04-06T02:44:10.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:10 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:10.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:10 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:44:10.844 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.858+0000 7f5720dd5700 1 -- 172.21.15.184:0/2461455243 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f571c105780 msgr2=0x7f571c105b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:44:10.844 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.858+0000 7f5720dd5700 1 --2- 172.21.15.184:0/2461455243 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f571c105780 0x7f571c105b60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f570c009910 tx=0x7f570c009c20 comp rx=0 tx=0).stop 2024-04-06T02:44:10.844 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.858+0000 7f5720dd5700 1 -- 172.21.15.184:0/2461455243 shutdown_connections 2024-04-06T02:44:10.844 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.858+0000 7f5720dd5700 1 --2- 172.21.15.184:0/2461455243 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f571c105780 0x7f571c105b60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:10.844 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.858+0000 7f5720dd5700 1 -- 172.21.15.184:0/2461455243 >> 172.21.15.184:0/2461455243 conn(0x7f571c0752b0 msgr2=0x7f571c0756c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:44:10.844 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.858+0000 7f5720dd5700 1 -- 172.21.15.184:0/2461455243 shutdown_connections 2024-04-06T02:44:10.844 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.858+0000 7f5720dd5700 1 -- 172.21.15.184:0/2461455243 wait complete. 2024-04-06T02:44:10.844 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.859+0000 7f5720dd5700 1 Processor -- start 2024-04-06T02:44:10.844 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.859+0000 7f5720dd5700 1 -- start start 2024-04-06T02:44:10.844 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.859+0000 7f5720dd5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f571c105780 0x7f571c196be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:44:10.845 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.859+0000 7f5720dd5700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571c197120 con 0x7f571c105780 2024-04-06T02:44:10.846 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.859+0000 7f571a59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f571c105780 0x7f571c196be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:44:10.846 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.859+0000 7f571a59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f571c105780 0x7f571c196be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.184:41552/0 (socket says 172.21.15.184:41552) 2024-04-06T02:44:10.846 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.859+0000 7f571a59c700 1 -- 172.21.15.184:0/1543136457 learned_addr learned my addr 172.21.15.184:0/1543136457 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-04-06T02:44:10.846 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.860+0000 7f571a59c700 1 -- 172.21.15.184:0/1543136457 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f570c0095f0 con 0x7f571c105780 2024-04-06T02:44:10.846 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.860+0000 7f571a59c700 1 --2- 172.21.15.184:0/1543136457 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f571c105780 0x7f571c196be0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f570c005b40 tx=0x7f570c004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:44:10.847 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.860+0000 7f570b7fe700 1 -- 172.21.15.184:0/1543136457 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f570c01d070 con 0x7f571c105780 2024-04-06T02:44:10.847 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.860+0000 7f570b7fe700 1 -- 172.21.15.184:0/1543136457 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f570c0044d0 con 0x7f571c105780 2024-04-06T02:44:10.847 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.860+0000 7f5720dd5700 1 -- 172.21.15.184:0/1543136457 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f571c1972c0 con 0x7f571c105780 2024-04-06T02:44:10.847 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.860+0000 7f570b7fe700 1 -- 172.21.15.184:0/1543136457 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f570c003930 con 0x7f571c105780 2024-04-06T02:44:10.847 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.860+0000 7f5720dd5700 1 -- 172.21.15.184:0/1543136457 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f571c197680 con 0x7f571c105780 2024-04-06T02:44:10.847 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.861+0000 7f570b7fe700 1 -- 172.21.15.184:0/1543136457 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7f570c003bf0 con 0x7f571c105780 2024-04-06T02:44:10.847 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.861+0000 7f5720dd5700 1 -- 172.21.15.184:0/1543136457 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f571c1039c0 con 0x7f571c105780 2024-04-06T02:44:10.847 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.861+0000 7f570b7fe700 1 --2- 172.21.15.184:0/1543136457 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f57040384b0 0x7f570403a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:44:10.847 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.861+0000 7f570b7fe700 1 -- 172.21.15.184:0/1543136457 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f570c04c420 con 0x7f571c105780 2024-04-06T02:44:10.847 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.861+0000 7f5719d9b700 1 --2- 172.21.15.184:0/1543136457 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f57040384b0 0x7f570403a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:44:10.847 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.862+0000 7f5719d9b700 1 --2- 172.21.15.184:0/1543136457 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f57040384b0 0x7f570403a970 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5710006fd0 tx=0x7f5710006e40 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:44:10.851 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:10.867+0000 7f570b7fe700 1 -- 172.21.15.184:0/1543136457 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f570c02a950 con 0x7f571c105780 2024-04-06T02:44:11.162 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:11.176+0000 7f5720dd5700 1 -- 172.21.15.184:0/1543136457 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f571c197a00 con 0x7f571c105780 2024-04-06T02:44:11.162 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:11.177+0000 7f570b7fe700 1 -- 172.21.15.184:0/1543136457 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f570c02a360 con 0x7f571c105780 2024-04-06T02:44:11.162 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:44:11.163 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"130df578-f3bf-11ee-b649-cb9ed24678a4","modified":"2024-04-06T02:41:29.716000Z","created":"2024-04-06T02:41:29.716000Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T02:44:11.164 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:11.179+0000 7f5720dd5700 1 -- 172.21.15.184:0/1543136457 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f57040384b0 msgr2=0x7f570403a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:44:11.164 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:11.179+0000 7f5720dd5700 1 --2- 172.21.15.184:0/1543136457 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f57040384b0 0x7f570403a970 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5710006fd0 tx=0x7f5710006e40 comp rx=0 tx=0).stop 2024-04-06T02:44:11.165 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:11.179+0000 7f5720dd5700 1 -- 172.21.15.184:0/1543136457 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f571c105780 msgr2=0x7f571c196be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:44:11.165 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:11.179+0000 7f5720dd5700 1 --2- 172.21.15.184:0/1543136457 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f571c105780 0x7f571c196be0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f570c005b40 tx=0x7f570c004c80 comp rx=0 tx=0).stop 2024-04-06T02:44:11.165 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:11.179+0000 7f5720dd5700 1 -- 172.21.15.184:0/1543136457 shutdown_connections 2024-04-06T02:44:11.165 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:11.179+0000 7f5720dd5700 1 --2- 172.21.15.184:0/1543136457 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f57040384b0 0x7f570403a970 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:11.165 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:11.179+0000 7f5720dd5700 1 --2- 172.21.15.184:0/1543136457 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f571c105780 0x7f571c196be0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:11.165 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:11.179+0000 7f5720dd5700 1 -- 172.21.15.184:0/1543136457 >> 172.21.15.184:0/1543136457 conn(0x7f571c0752b0 msgr2=0x7f571c0fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:44:11.165 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:11.179+0000 7f5720dd5700 1 -- 172.21.15.184:0/1543136457 shutdown_connections 2024-04-06T02:44:11.165 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:11.179+0000 7f5720dd5700 1 -- 172.21.15.184:0/1543136457 wait complete. 2024-04-06T02:44:11.165 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-04-06T02:44:11.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:11 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/1543136457' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T02:44:12.781 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T02:44:12.781 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T02:44:13.207 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:44:14.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:14 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:14.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:14 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:14.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:14 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:14.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:14 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:14.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:14 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi184.fzhhql", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:44:14.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:14 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi184.fzhhql", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-06T02:44:14.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:14 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:44:14.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:14 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:44:14.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:14 smithi105 ceph-mon[29228]: Deploying daemon mgr.smithi184.fzhhql on smithi184 2024-04-06T02:44:14.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:14 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:15.094 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.110+0000 7f7b7d0ca700 1 -- 172.21.15.184:0/1269635861 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b78101ab0 msgr2=0x7f7b78101e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:44:15.095 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.110+0000 7f7b7d0ca700 1 --2- 172.21.15.184:0/1269635861 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b78101ab0 0x7f7b78101e90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7b7000b0a0 tx=0x7f7b7000b3b0 comp rx=0 tx=0).stop 2024-04-06T02:44:15.095 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.110+0000 7f7b7d0ca700 1 -- 172.21.15.184:0/1269635861 shutdown_connections 2024-04-06T02:44:15.095 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.110+0000 7f7b7d0ca700 1 --2- 172.21.15.184:0/1269635861 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b78101ab0 0x7f7b78101e90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:15.095 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.110+0000 7f7b7d0ca700 1 -- 172.21.15.184:0/1269635861 >> 172.21.15.184:0/1269635861 conn(0x7f7b780fb240 msgr2=0x7f7b780fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:44:15.095 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.110+0000 7f7b7d0ca700 1 -- 172.21.15.184:0/1269635861 shutdown_connections 2024-04-06T02:44:15.095 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.110+0000 7f7b7d0ca700 1 -- 172.21.15.184:0/1269635861 wait complete. 2024-04-06T02:44:15.095 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.111+0000 7f7b7d0ca700 1 Processor -- start 2024-04-06T02:44:15.095 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.111+0000 7f7b7d0ca700 1 -- start start 2024-04-06T02:44:15.096 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.111+0000 7f7b7d0ca700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b78101ab0 0x7f7b78195dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:44:15.096 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.111+0000 7f7b7d0ca700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b78196310 con 0x7f7b78101ab0 2024-04-06T02:44:15.097 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.111+0000 7f7b76d9d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b78101ab0 0x7f7b78195dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:44:15.097 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.111+0000 7f7b76d9d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b78101ab0 0x7f7b78195dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.184:41574/0 (socket says 172.21.15.184:41574) 2024-04-06T02:44:15.097 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.111+0000 7f7b76d9d700 1 -- 172.21.15.184:0/3286994212 learned_addr learned my addr 172.21.15.184:0/3286994212 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-04-06T02:44:15.097 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.112+0000 7f7b76d9d700 1 -- 172.21.15.184:0/3286994212 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b70009d00 con 0x7f7b78101ab0 2024-04-06T02:44:15.097 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.112+0000 7f7b76d9d700 1 --2- 172.21.15.184:0/3286994212 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b78101ab0 0x7f7b78195dd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7b70009fd0 tx=0x7f7b70009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:44:15.097 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.112+0000 7f7b6ffff700 1 -- 172.21.15.184:0/3286994212 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b7000e040 con 0x7f7b78101ab0 2024-04-06T02:44:15.097 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.112+0000 7f7b7d0ca700 1 -- 172.21.15.184:0/3286994212 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b78196510 con 0x7f7b78101ab0 2024-04-06T02:44:15.097 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.112+0000 7f7b6ffff700 1 -- 172.21.15.184:0/3286994212 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7b70004040 con 0x7f7b78101ab0 2024-04-06T02:44:15.097 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.112+0000 7f7b6ffff700 1 -- 172.21.15.184:0/3286994212 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b7001d440 con 0x7f7b78101ab0 2024-04-06T02:44:15.097 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.112+0000 7f7b7d0ca700 1 -- 172.21.15.184:0/3286994212 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b78196990 con 0x7f7b78101ab0 2024-04-06T02:44:15.098 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.113+0000 7f7b7d0ca700 1 -- 172.21.15.184:0/3286994212 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b780623c0 con 0x7f7b78101ab0 2024-04-06T02:44:15.098 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.113+0000 7f7b6ffff700 1 -- 172.21.15.184:0/3286994212 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7f7b700045b0 con 0x7f7b78101ab0 2024-04-06T02:44:15.098 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.113+0000 7f7b6ffff700 1 --2- 172.21.15.184:0/3286994212 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f7b640384b0 0x7f7b6403a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:44:15.098 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.113+0000 7f7b6ffff700 1 -- 172.21.15.184:0/3286994212 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f7b7004bff0 con 0x7f7b78101ab0 2024-04-06T02:44:15.098 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.114+0000 7f7b7659c700 1 --2- 172.21.15.184:0/3286994212 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f7b640384b0 0x7f7b6403a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:44:15.099 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.114+0000 7f7b7659c700 1 --2- 172.21.15.184:0/3286994212 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f7b640384b0 0x7f7b6403a970 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7b60006fd0 tx=0x7f7b60006e40 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:44:15.102 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.118+0000 7f7b6ffff700 1 -- 172.21.15.184:0/3286994212 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7b70011960 con 0x7f7b78101ab0 2024-04-06T02:44:15.374 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.389+0000 7f7b7d0ca700 1 -- 172.21.15.184:0/3286994212 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b78107ab0 con 0x7f7b78101ab0 2024-04-06T02:44:15.374 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.390+0000 7f7b6ffff700 1 -- 172.21.15.184:0/3286994212 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f7b70011390 con 0x7f7b78101ab0 2024-04-06T02:44:15.374 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:44:15.374 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"130df578-f3bf-11ee-b649-cb9ed24678a4","modified":"2024-04-06T02:41:29.716000Z","created":"2024-04-06T02:41:29.716000Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T02:44:15.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.392+0000 7f7b7d0ca700 1 -- 172.21.15.184:0/3286994212 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f7b640384b0 msgr2=0x7f7b6403a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:44:15.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.392+0000 7f7b7d0ca700 1 --2- 172.21.15.184:0/3286994212 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f7b640384b0 0x7f7b6403a970 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7b60006fd0 tx=0x7f7b60006e40 comp rx=0 tx=0).stop 2024-04-06T02:44:15.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.392+0000 7f7b7d0ca700 1 -- 172.21.15.184:0/3286994212 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b78101ab0 msgr2=0x7f7b78195dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:44:15.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.392+0000 7f7b7d0ca700 1 --2- 172.21.15.184:0/3286994212 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b78101ab0 0x7f7b78195dd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7b70009fd0 tx=0x7f7b70009800 comp rx=0 tx=0).stop 2024-04-06T02:44:15.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.392+0000 7f7b7d0ca700 1 -- 172.21.15.184:0/3286994212 shutdown_connections 2024-04-06T02:44:15.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.392+0000 7f7b7d0ca700 1 --2- 172.21.15.184:0/3286994212 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f7b640384b0 0x7f7b6403a970 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:15.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.392+0000 7f7b7d0ca700 1 --2- 172.21.15.184:0/3286994212 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b78101ab0 0x7f7b78195dd0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:15.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.392+0000 7f7b7d0ca700 1 -- 172.21.15.184:0/3286994212 >> 172.21.15.184:0/3286994212 conn(0x7f7b780fb240 msgr2=0x7f7b78105100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:44:15.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.392+0000 7f7b7d0ca700 1 -- 172.21.15.184:0/3286994212 shutdown_connections 2024-04-06T02:44:15.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:15.392+0000 7f7b7d0ca700 1 -- 172.21.15.184:0/3286994212 wait complete. 2024-04-06T02:44:15.377 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-04-06T02:44:16.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:15 smithi105 ceph-mon[29228]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:16.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:15 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/3286994212' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T02:44:17.745 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T02:44:17.746 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T02:44:18.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:17 smithi105 ceph-mon[29228]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:18.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:17 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:18.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:17 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:18.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:17 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:18.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:17 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:18.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:17 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:44:18.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:17 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:44:18.721 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:44:19.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:18 smithi105 ceph-mon[29228]: Deploying daemon mon.smithi184 on smithi184 2024-04-06T02:44:20.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:19 smithi105 ceph-mon[29228]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:20.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:19 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:21.414 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.428+0000 7fa7b8508700 1 -- 172.21.15.184:0/243023689 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7b0106810 msgr2=0x7fa7b0106bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:44:21.415 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.428+0000 7fa7b8508700 1 --2- 172.21.15.184:0/243023689 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7b0106810 0x7fa7b0106bf0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac009910 tx=0x7fa7ac009c20 comp rx=0 tx=0).stop 2024-04-06T02:44:21.415 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.429+0000 7fa7b8508700 1 -- 172.21.15.184:0/243023689 shutdown_connections 2024-04-06T02:44:21.415 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.429+0000 7fa7b8508700 1 --2- 172.21.15.184:0/243023689 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7b0106810 0x7fa7b0106bf0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:21.415 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.429+0000 7fa7b8508700 1 -- 172.21.15.184:0/243023689 >> 172.21.15.184:0/243023689 conn(0x7fa7b00fc320 msgr2=0x7fa7b00fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:44:21.415 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.429+0000 7fa7b8508700 1 -- 172.21.15.184:0/243023689 shutdown_connections 2024-04-06T02:44:21.415 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.429+0000 7fa7b8508700 1 -- 172.21.15.184:0/243023689 wait complete. 2024-04-06T02:44:21.415 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.429+0000 7fa7b8508700 1 Processor -- start 2024-04-06T02:44:21.415 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.429+0000 7fa7b8508700 1 -- start start 2024-04-06T02:44:21.419 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.430+0000 7fa7b8508700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7b0106810 0x7fa7b0195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:44:21.419 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.430+0000 7fa7b8508700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b01962a0 con 0x7fa7b0106810 2024-04-06T02:44:21.419 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.430+0000 7fa7b62a4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7b0106810 0x7fa7b0195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:44:21.419 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.430+0000 7fa7b62a4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7b0106810 0x7fa7b0195d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.184:60612/0 (socket says 172.21.15.184:60612) 2024-04-06T02:44:21.419 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.430+0000 7fa7b62a4700 1 -- 172.21.15.184:0/583723278 learned_addr learned my addr 172.21.15.184:0/583723278 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-04-06T02:44:21.419 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.431+0000 7fa7b62a4700 1 -- 172.21.15.184:0/583723278 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7ac0095f0 con 0x7fa7b0106810 2024-04-06T02:44:21.419 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.431+0000 7fa7b62a4700 1 --2- 172.21.15.184:0/583723278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7b0106810 0x7fa7b0195d60 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac005b40 tx=0x7fa7ac004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:44:21.419 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.431+0000 7fa7a77fe700 1 -- 172.21.15.184:0/583723278 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7ac01d070 con 0x7fa7b0106810 2024-04-06T02:44:21.419 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.431+0000 7fa7a77fe700 1 -- 172.21.15.184:0/583723278 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa7ac0044d0 con 0x7fa7b0106810 2024-04-06T02:44:21.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.431+0000 7fa7b8508700 1 -- 172.21.15.184:0/583723278 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b01964a0 con 0x7fa7b0106810 2024-04-06T02:44:21.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.431+0000 7fa7a77fe700 1 -- 172.21.15.184:0/583723278 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7ac003910 con 0x7fa7b0106810 2024-04-06T02:44:21.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.432+0000 7fa7b8508700 1 -- 172.21.15.184:0/583723278 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b0196920 con 0x7fa7b0106810 2024-04-06T02:44:21.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.432+0000 7fa7a77fe700 1 -- 172.21.15.184:0/583723278 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7fa7ac003b90 con 0x7fa7b0106810 2024-04-06T02:44:21.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.433+0000 7fa7b8508700 1 -- 172.21.15.184:0/583723278 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7b00623c0 con 0x7fa7b0106810 2024-04-06T02:44:21.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.433+0000 7fa7a77fe700 1 --2- 172.21.15.184:0/583723278 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fa79c0384f0 0x7fa79c03a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:44:21.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.433+0000 7fa7a77fe700 1 -- 172.21.15.184:0/583723278 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fa7ac00b960 con 0x7fa7b0106810 2024-04-06T02:44:21.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.433+0000 7fa7b5aa3700 1 --2- 172.21.15.184:0/583723278 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fa79c0384f0 0x7fa79c03a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:44:21.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.434+0000 7fa7b5aa3700 1 --2- 172.21.15.184:0/583723278 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fa79c0384f0 0x7fa79c03a9b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0006fd0 tx=0x7fa7a0006e40 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:44:21.423 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.438+0000 7fa7a77fe700 1 -- 172.21.15.184:0/583723278 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa7ac02a950 con 0x7fa7b0106810 2024-04-06T02:44:21.700 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.715+0000 7fa7b8508700 1 -- 172.21.15.184:0/583723278 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa7b01970c0 con 0x7fa7b0106810 2024-04-06T02:44:21.700 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.715+0000 7fa7a77fe700 1 -- 172.21.15.184:0/583723278 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fa7ac02a950 con 0x7fa7b0106810 2024-04-06T02:44:21.700 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:44:21.701 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"130df578-f3bf-11ee-b649-cb9ed24678a4","modified":"2024-04-06T02:41:29.716000Z","created":"2024-04-06T02:41:29.716000Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T02:44:21.703 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.717+0000 7fa7b8508700 1 -- 172.21.15.184:0/583723278 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fa79c0384f0 msgr2=0x7fa79c03a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:44:21.703 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.717+0000 7fa7b8508700 1 --2- 172.21.15.184:0/583723278 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fa79c0384f0 0x7fa79c03a9b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0006fd0 tx=0x7fa7a0006e40 comp rx=0 tx=0).stop 2024-04-06T02:44:21.703 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.717+0000 7fa7b8508700 1 -- 172.21.15.184:0/583723278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7b0106810 msgr2=0x7fa7b0195d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:44:21.703 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.717+0000 7fa7b8508700 1 --2- 172.21.15.184:0/583723278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7b0106810 0x7fa7b0195d60 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac005b40 tx=0x7fa7ac004c80 comp rx=0 tx=0).stop 2024-04-06T02:44:21.703 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.718+0000 7fa7b8508700 1 -- 172.21.15.184:0/583723278 shutdown_connections 2024-04-06T02:44:21.703 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.718+0000 7fa7b8508700 1 --2- 172.21.15.184:0/583723278 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fa79c0384f0 0x7fa79c03a9b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:21.703 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.718+0000 7fa7b8508700 1 --2- 172.21.15.184:0/583723278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7b0106810 0x7fa7b0195d60 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:21.703 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.718+0000 7fa7b8508700 1 -- 172.21.15.184:0/583723278 >> 172.21.15.184:0/583723278 conn(0x7fa7b00fc320 msgr2=0x7fa7b0108bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:44:21.703 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.718+0000 7fa7b8508700 1 -- 172.21.15.184:0/583723278 shutdown_connections 2024-04-06T02:44:21.703 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:21.718+0000 7fa7b8508700 1 -- 172.21.15.184:0/583723278 wait complete. 2024-04-06T02:44:21.704 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-04-06T02:44:21.999 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:21 smithi105 ceph-mon[29228]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:21.999 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:21 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/583723278' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T02:44:22.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:22 smithi184 systemd[1]: Started Ceph mon.smithi184 for 130df578-f3bf-11ee-b649-cb9ed24678a4. 2024-04-06T02:44:23.398 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-06T02:44:23.398 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T02:44:23.613 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi184/config 2024-04-06T02:44:27.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-04-06T02:44:27.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: mon.smithi105 calling monitor election 2024-04-06T02:44:27.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:27.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-04-06T02:44:27.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-04-06T02:44:27.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: from='mgr.? 172.21.15.184:0/3121118228' entity='mgr.smithi184.fzhhql' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi184.fzhhql/crt"}]: dispatch 2024-04-06T02:44:27.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-04-06T02:44:27.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: mon.smithi184 calling monitor election 2024-04-06T02:44:27.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:27.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:44:27.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-04-06T02:44:27.926 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-04-06T02:44:27.926 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:27.926 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-04-06T02:44:27.926 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: mon.smithi105 is new leader, mons smithi105,smithi184 in quorum (ranks 0,1) 2024-04-06T02:44:27.926 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: monmap e2: 2 mons at {smithi105=[v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0],smithi184=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-06T02:44:27.926 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: fsmap 2024-04-06T02:44:27.926 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: osdmap e5: 0 total, 0 up, 0 in 2024-04-06T02:44:27.926 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: mgrmap e17: smithi105.vqqcpx(active, since 33s) 2024-04-06T02:44:27.926 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: Standby manager daemon smithi184.fzhhql started 2024-04-06T02:44:27.926 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: from='mgr.? 172.21.15.184:0/3121118228' entity='mgr.smithi184.fzhhql' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T02:44:27.926 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: overall HEALTH_OK 2024-04-06T02:44:27.926 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: from='mgr.? 172.21.15.184:0/3121118228' entity='mgr.smithi184.fzhhql' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi184.fzhhql/key"}]: dispatch 2024-04-06T02:44:27.926 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: from='mgr.? 172.21.15.184:0/3121118228' entity='mgr.smithi184.fzhhql' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T02:44:27.926 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:27.926 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:27.927 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:27.927 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:27.927 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:27.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-04-06T02:44:27.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: mon.smithi105 calling monitor election 2024-04-06T02:44:27.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:27.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-04-06T02:44:27.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-04-06T02:44:27.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: from='mgr.? 172.21.15.184:0/3121118228' entity='mgr.smithi184.fzhhql' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi184.fzhhql/crt"}]: dispatch 2024-04-06T02:44:27.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-04-06T02:44:27.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: mon.smithi184 calling monitor election 2024-04-06T02:44:27.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:27.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:44:27.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-04-06T02:44:27.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-04-06T02:44:27.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:27.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-04-06T02:44:27.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: mon.smithi105 is new leader, mons smithi105,smithi184 in quorum (ranks 0,1) 2024-04-06T02:44:27.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: monmap e2: 2 mons at {smithi105=[v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0],smithi184=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-06T02:44:27.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: fsmap 2024-04-06T02:44:27.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: osdmap e5: 0 total, 0 up, 0 in 2024-04-06T02:44:27.982 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: mgrmap e17: smithi105.vqqcpx(active, since 33s) 2024-04-06T02:44:27.982 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: Standby manager daemon smithi184.fzhhql started 2024-04-06T02:44:27.982 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: from='mgr.? 172.21.15.184:0/3121118228' entity='mgr.smithi184.fzhhql' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T02:44:27.982 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: overall HEALTH_OK 2024-04-06T02:44:27.982 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: from='mgr.? 172.21.15.184:0/3121118228' entity='mgr.smithi184.fzhhql' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi184.fzhhql/key"}]: dispatch 2024-04-06T02:44:27.982 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: from='mgr.? 172.21.15.184:0/3121118228' entity='mgr.smithi184.fzhhql' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T02:44:27.982 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:27.982 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:27.982 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:27.982 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:27.982 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:27.984 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.995+0000 7f170a6e0700 1 -- 172.21.15.184:0/824170381 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f16ec0035c0 msgr2=0x7f16ec005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:44:27.984 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.995+0000 7f170a6e0700 1 --2- 172.21.15.184:0/824170381 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f16ec0035c0 0x7f16ec005a50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f17040ff790 tx=0x7f16f400feb0 comp rx=0 tx=0).stop 2024-04-06T02:44:27.984 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.996+0000 7f170a6e0700 1 -- 172.21.15.184:0/824170381 shutdown_connections 2024-04-06T02:44:27.984 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.996+0000 7f170a6e0700 1 --2- 172.21.15.184:0/824170381 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f16ec0035c0 0x7f16ec005a50 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:27.984 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.996+0000 7f170a6e0700 1 -- 172.21.15.184:0/824170381 >> 172.21.15.184:0/824170381 conn(0x7f17040fb390 msgr2=0x7f17040fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:44:27.984 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.996+0000 7f170a6e0700 1 -- 172.21.15.184:0/824170381 shutdown_connections 2024-04-06T02:44:27.985 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.996+0000 7f170a6e0700 1 -- 172.21.15.184:0/824170381 wait complete. 2024-04-06T02:44:27.985 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.996+0000 7f170a6e0700 1 Processor -- start 2024-04-06T02:44:27.985 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.996+0000 7f170a6e0700 1 -- start start 2024-04-06T02:44:27.985 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.997+0000 7f170a6e0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f16ec0035c0 0x7f1704101430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:44:27.985 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.997+0000 7f170a6e0700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1704101970 0x7f1704101df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:44:27.985 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.997+0000 7f170a6e0700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17041a20a0 con 0x7f16ec0035c0 2024-04-06T02:44:27.985 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.997+0000 7f170a6e0700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17041a21e0 con 0x7f1704101970 2024-04-06T02:44:27.985 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.997+0000 7f17037fe700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1704101970 0x7f1704101df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:44:27.985 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.997+0000 7f17037fe700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1704101970 0x7f1704101df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:52876/0 (socket says 172.21.15.184:52876) 2024-04-06T02:44:27.985 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.997+0000 7f17037fe700 1 -- 172.21.15.184:0/1504599588 learned_addr learned my addr 172.21.15.184:0/1504599588 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-04-06T02:44:27.985 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.997+0000 7f1703fff700 1 --2- 172.21.15.184:0/1504599588 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f16ec0035c0 0x7f1704101430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:44:27.985 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.997+0000 7f17037fe700 1 -- 172.21.15.184:0/1504599588 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1704101970 msgr2=0x7f1704101df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-06T02:44:27.985 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.997+0000 7f17037fe700 1 -- 172.21.15.184:0/1504599588 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1704101970 msgr2=0x7f1704101df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-06T02:44:27.985 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.997+0000 7f17037fe700 1 --2- 172.21.15.184:0/1504599588 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1704101970 0x7f1704101df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-06T02:44:27.985 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.997+0000 7f17037fe700 1 --2- 172.21.15.184:0/1504599588 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1704101970 0x7f1704101df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-06T02:44:27.986 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.997+0000 7f1703fff700 1 -- 172.21.15.184:0/1504599588 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1704101970 msgr2=0x7f1704101df0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-06T02:44:27.986 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.997+0000 7f1703fff700 1 --2- 172.21.15.184:0/1504599588 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1704101970 0x7f1704101df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:27.986 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.997+0000 7f1703fff700 1 -- 172.21.15.184:0/1504599588 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f400f900 con 0x7f16ec0035c0 2024-04-06T02:44:27.986 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.998+0000 7f1703fff700 1 --2- 172.21.15.184:0/1504599588 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f16ec0035c0 0x7f1704101430 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f17040ff790 tx=0x7f16f4004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:44:27.986 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.998+0000 7f17017fa700 1 -- 172.21.15.184:0/1504599588 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16f4004b50 con 0x7f16ec0035c0 2024-04-06T02:44:27.986 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.998+0000 7f17017fa700 1 -- 172.21.15.184:0/1504599588 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f16f4013e10 con 0x7f16ec0035c0 2024-04-06T02:44:27.986 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.998+0000 7f170a6e0700 1 -- 172.21.15.184:0/1504599588 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17041a2320 con 0x7f16ec0035c0 2024-04-06T02:44:27.986 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.998+0000 7f17017fa700 1 -- 172.21.15.184:0/1504599588 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16f4011cd0 con 0x7f16ec0035c0 2024-04-06T02:44:27.986 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.998+0000 7f170a6e0700 1 -- 172.21.15.184:0/1504599588 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17041a2810 con 0x7f16ec0035c0 2024-04-06T02:44:27.986 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:27.999+0000 7f170a6e0700 1 -- 172.21.15.184:0/1504599588 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17040623c0 con 0x7f16ec0035c0 2024-04-06T02:44:27.987 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:28.000+0000 7f17017fa700 1 -- 172.21.15.184:0/1504599588 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7f16f4003680 con 0x7f16ec0035c0 2024-04-06T02:44:27.987 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:28.000+0000 7f17017fa700 1 --2- 172.21.15.184:0/1504599588 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f16f006c5b0 0x7f16f006ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:44:27.987 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:28.000+0000 7f17017fa700 1 -- 172.21.15.184:0/1504599588 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f16f408cfb0 con 0x7f16ec0035c0 2024-04-06T02:44:27.987 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:28.000+0000 7f17037fe700 1 --2- 172.21.15.184:0/1504599588 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f16f006c5b0 0x7f16f006ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:44:27.987 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:28.001+0000 7f17037fe700 1 --2- 172.21.15.184:0/1504599588 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f16f006c5b0 0x7f16f006ea70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f16f8009180 tx=0x7f16f8008040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:44:27.990 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:28.005+0000 7f17017fa700 1 -- 172.21.15.184:0/1504599588 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f16f40c00e0 con 0x7f16ec0035c0 2024-04-06T02:44:28.314 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:28.329+0000 7f170a6e0700 1 -- 172.21.15.184:0/1504599588 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1704102960 con 0x7f16ec0035c0 2024-04-06T02:44:28.315 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:28.330+0000 7f17017fa700 1 -- 172.21.15.184:0/1504599588 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1026 (secure 0 0 0) 0x7f16f40589d0 con 0x7f16ec0035c0 2024-04-06T02:44:28.315 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-04-06T02:44:28.315 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":2,"fsid":"130df578-f3bf-11ee-b649-cb9ed24678a4","modified":"2024-04-06T02:44:22.575713Z","created":"2024-04-06T02:41:29.716000Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi184","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:3300","nonce":0},{"type":"v1","addr":"172.21.15.184:6789","nonce":0}]},"addr":"172.21.15.184:6789/0","public_addr":"172.21.15.184:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-06T02:44:28.317 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:28.332+0000 7f170a6e0700 1 -- 172.21.15.184:0/1504599588 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f16f006c5b0 msgr2=0x7f16f006ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:44:28.317 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:28.332+0000 7f170a6e0700 1 --2- 172.21.15.184:0/1504599588 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f16f006c5b0 0x7f16f006ea70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f16f8009180 tx=0x7f16f8008040 comp rx=0 tx=0).stop 2024-04-06T02:44:28.317 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:28.332+0000 7f170a6e0700 1 -- 172.21.15.184:0/1504599588 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f16ec0035c0 msgr2=0x7f1704101430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:44:28.318 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:28.332+0000 7f170a6e0700 1 --2- 172.21.15.184:0/1504599588 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f16ec0035c0 0x7f1704101430 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f17040ff790 tx=0x7f16f4004f90 comp rx=0 tx=0).stop 2024-04-06T02:44:28.318 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:28.332+0000 7f170a6e0700 1 -- 172.21.15.184:0/1504599588 shutdown_connections 2024-04-06T02:44:28.318 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:28.332+0000 7f170a6e0700 1 --2- 172.21.15.184:0/1504599588 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f16f006c5b0 0x7f16f006ea70 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:28.318 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:28.332+0000 7f170a6e0700 1 --2- 172.21.15.184:0/1504599588 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f16ec0035c0 0x7f1704101430 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:28.318 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:28.332+0000 7f170a6e0700 1 --2- 172.21.15.184:0/1504599588 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1704101970 0x7f1704101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:28.318 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:28.332+0000 7f170a6e0700 1 -- 172.21.15.184:0/1504599588 >> 172.21.15.184:0/1504599588 conn(0x7f17040fb390 msgr2=0x7f1704068d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:44:28.318 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:28.332+0000 7f170a6e0700 1 -- 172.21.15.184:0/1504599588 shutdown_connections 2024-04-06T02:44:28.318 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:44:28.332+0000 7f170a6e0700 1 -- 172.21.15.184:0/1504599588 wait complete. 2024-04-06T02:44:28.318 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 2 2024-04-06T02:44:28.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:28 smithi105 ceph-mon[29228]: mgrmap e18: smithi105.vqqcpx(active, since 33s), standbys: smithi184.fzhhql 2024-04-06T02:44:28.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:28 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mgr metadata", "who": "smithi184.fzhhql", "id": "smithi184.fzhhql"}]: dispatch 2024-04-06T02:44:28.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:28 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:44:28.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:28 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/1504599588' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T02:44:28.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:28 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-04-06T02:44:28.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:28 smithi184 ceph-mon[37439]: mgrmap e18: smithi105.vqqcpx(active, since 33s), standbys: smithi184.fzhhql 2024-04-06T02:44:28.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:28 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mgr metadata", "who": "smithi184.fzhhql", "id": "smithi184.fzhhql"}]: dispatch 2024-04-06T02:44:28.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:28 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:44:28.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:28 smithi184 ceph-mon[37439]: from='client.? 172.21.15.184:0/1504599588' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T02:44:28.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:28 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-04-06T02:44:29.011 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-06T02:44:29.011 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph config generate-minimal-conf 2024-04-06T02:44:29.228 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:44:29.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:29 smithi105 ceph-mon[29228]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:29.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:29 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:29.954 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:29 smithi184 ceph-mon[37439]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:29.954 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:29 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:30.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.951+0000 7f0eead08700 1 -- 172.21.15.105:0/3151848649 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ee4101a30 msgr2=0x7f0ee4101e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:44:30.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.951+0000 7f0eead08700 1 --2- 172.21.15.105:0/3151848649 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ee4101a30 0x7f0ee4101e10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8009910 tx=0x7f0ed8009c20 comp rx=0 tx=0).stop 2024-04-06T02:44:30.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.951+0000 7f0eead08700 1 -- 172.21.15.105:0/3151848649 shutdown_connections 2024-04-06T02:44:30.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.951+0000 7f0eead08700 1 --2- 172.21.15.105:0/3151848649 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ee4101a30 0x7f0ee4101e10 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:30.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.951+0000 7f0eead08700 1 -- 172.21.15.105:0/3151848649 >> 172.21.15.105:0/3151848649 conn(0x7f0ee40fb220 msgr2=0x7f0ee40fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:44:30.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.951+0000 7f0eead08700 1 -- 172.21.15.105:0/3151848649 shutdown_connections 2024-04-06T02:44:30.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.951+0000 7f0eead08700 1 -- 172.21.15.105:0/3151848649 wait complete. 2024-04-06T02:44:30.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.951+0000 7f0eead08700 1 Processor -- start 2024-04-06T02:44:30.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.951+0000 7f0eead08700 1 -- start start 2024-04-06T02:44:30.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.952+0000 7f0eead08700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ee4101a30 0x7f0ee4195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:44:30.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.952+0000 7f0eead08700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ee4196420 0x7f0ee419a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:44:30.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.952+0000 7f0eead08700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee4196a40 con 0x7f0ee4101a30 2024-04-06T02:44:30.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.952+0000 7f0eead08700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee4196bb0 con 0x7f0ee4196420 2024-04-06T02:44:30.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.952+0000 7f0ee8aa4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ee4101a30 0x7f0ee4195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:44:30.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.952+0000 7f0ee8aa4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ee4101a30 0x7f0ee4195ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:49058/0 (socket says 172.21.15.105:49058) 2024-04-06T02:44:30.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.952+0000 7f0ee8aa4700 1 -- 172.21.15.105:0/1820937376 learned_addr learned my addr 172.21.15.105:0/1820937376 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:44:30.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.952+0000 7f0ee3fff700 1 --2- 172.21.15.105:0/1820937376 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ee4196420 0x7f0ee419a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:44:30.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.952+0000 7f0ee8aa4700 1 -- 172.21.15.105:0/1820937376 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ee4196420 msgr2=0x7f0ee419a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:44:30.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.952+0000 7f0ee8aa4700 1 --2- 172.21.15.105:0/1820937376 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ee4196420 0x7f0ee419a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:30.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.952+0000 7f0ee8aa4700 1 -- 172.21.15.105:0/1820937376 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed80095f0 con 0x7f0ee4101a30 2024-04-06T02:44:30.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.952+0000 7f0ee3fff700 1 --2- 172.21.15.105:0/1820937376 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ee4196420 0x7f0ee419a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T02:44:30.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.952+0000 7f0ee8aa4700 1 --2- 172.21.15.105:0/1820937376 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ee4101a30 0x7f0ee4195ee0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8006010 tx=0x7f0ed800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:44:30.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.952+0000 7f0ee1ffb700 1 -- 172.21.15.105:0/1820937376 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ed801d070 con 0x7f0ee4101a30 2024-04-06T02:44:30.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.952+0000 7f0ee1ffb700 1 -- 172.21.15.105:0/1820937376 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0ed8005140 con 0x7f0ee4101a30 2024-04-06T02:44:30.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.952+0000 7f0eead08700 1 -- 172.21.15.105:0/1820937376 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ee419ae30 con 0x7f0ee4101a30 2024-04-06T02:44:30.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.952+0000 7f0ee1ffb700 1 -- 172.21.15.105:0/1820937376 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ed8017440 con 0x7f0ee4101a30 2024-04-06T02:44:30.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.953+0000 7f0eead08700 1 -- 172.21.15.105:0/1820937376 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ee419b380 con 0x7f0ee4101a30 2024-04-06T02:44:30.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.954+0000 7f0eead08700 1 -- 172.21.15.105:0/1820937376 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ee4108120 con 0x7f0ee4101a30 2024-04-06T02:44:30.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.954+0000 7f0ee1ffb700 1 -- 172.21.15.105:0/1820937376 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7f0ed800f460 con 0x7f0ee4101a30 2024-04-06T02:44:30.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.955+0000 7f0ee1ffb700 1 --2- 172.21.15.105:0/1820937376 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f0ed406c5c0 0x7f0ed406ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:44:30.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.955+0000 7f0ee3fff700 1 --2- 172.21.15.105:0/1820937376 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f0ed406c5c0 0x7f0ed406ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:44:30.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.955+0000 7f0ee1ffb700 1 -- 172.21.15.105:0/1820937376 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f0ed808ba30 con 0x7f0ee4101a30 2024-04-06T02:44:30.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.955+0000 7f0ee3fff700 1 --2- 172.21.15.105:0/1820937376 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f0ed406c5c0 0x7f0ed406ea80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0ed00079b0 tx=0x7f0ed0008040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:44:30.944 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:30.959+0000 7f0ee1ffb700 1 -- 172.21.15.105:0/1820937376 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0ed80572d0 con 0x7f0ee4101a30 2024-04-06T02:44:31.145 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:31.160+0000 7f0eead08700 1 -- 172.21.15.105:0/1820937376 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f0ee40623c0 con 0x7f0ee4101a30 2024-04-06T02:44:31.145 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:31.161+0000 7f0ee1ffb700 1 -- 172.21.15.105:0/1820937376 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+227 (secure 0 0 0) 0x7f0ed8026090 con 0x7f0ee4101a30 2024-04-06T02:44:31.146 INFO:teuthology.orchestra.run.smithi105.stdout:# minimal ceph.conf for 130df578-f3bf-11ee-b649-cb9ed24678a4 2024-04-06T02:44:31.146 INFO:teuthology.orchestra.run.smithi105.stdout:[global] 2024-04-06T02:44:31.146 INFO:teuthology.orchestra.run.smithi105.stdout: fsid = 130df578-f3bf-11ee-b649-cb9ed24678a4 2024-04-06T02:44:31.146 INFO:teuthology.orchestra.run.smithi105.stdout: mon_host = [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] 2024-04-06T02:44:31.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:31.163+0000 7f0eead08700 1 -- 172.21.15.105:0/1820937376 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f0ed406c5c0 msgr2=0x7f0ed406ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:44:31.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:31.163+0000 7f0eead08700 1 --2- 172.21.15.105:0/1820937376 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f0ed406c5c0 0x7f0ed406ea80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0ed00079b0 tx=0x7f0ed0008040 comp rx=0 tx=0).stop 2024-04-06T02:44:31.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:31.163+0000 7f0eead08700 1 -- 172.21.15.105:0/1820937376 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ee4101a30 msgr2=0x7f0ee4195ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:44:31.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:31.163+0000 7f0eead08700 1 --2- 172.21.15.105:0/1820937376 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ee4101a30 0x7f0ee4195ee0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8006010 tx=0x7f0ed800bda0 comp rx=0 tx=0).stop 2024-04-06T02:44:31.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:31.163+0000 7f0eead08700 1 -- 172.21.15.105:0/1820937376 shutdown_connections 2024-04-06T02:44:31.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:31.163+0000 7f0eead08700 1 --2- 172.21.15.105:0/1820937376 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f0ed406c5c0 0x7f0ed406ea80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:31.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:31.163+0000 7f0eead08700 1 --2- 172.21.15.105:0/1820937376 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ee4101a30 0x7f0ee4195ee0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:31.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:31.163+0000 7f0eead08700 1 --2- 172.21.15.105:0/1820937376 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ee4196420 0x7f0ee419a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:31.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:31.163+0000 7f0eead08700 1 -- 172.21.15.105:0/1820937376 >> 172.21.15.105:0/1820937376 conn(0x7f0ee40fb220 msgr2=0x7f0ee4105260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:44:31.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:31.163+0000 7f0eead08700 1 -- 172.21.15.105:0/1820937376 shutdown_connections 2024-04-06T02:44:31.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:31.163+0000 7f0eead08700 1 -- 172.21.15.105:0/1820937376 wait complete. 2024-04-06T02:44:31.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:31 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:31.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:31 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:31.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:31 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:44:31.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:31 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:44:31.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:31 smithi184 ceph-mon[37439]: Updating smithi105:/etc/ceph/ceph.conf 2024-04-06T02:44:31.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:31 smithi184 ceph-mon[37439]: Updating smithi184:/etc/ceph/ceph.conf 2024-04-06T02:44:31.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:31 smithi184 ceph-mon[37439]: Updating smithi105:/var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:44:31.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:31 smithi184 ceph-mon[37439]: Updating smithi184:/var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:44:31.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:31 smithi184 ceph-mon[37439]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:31.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:31 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:31.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:31 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:31.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:31 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:31.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:31 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:31.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:31 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:31.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:31 smithi184 ceph-mon[37439]: Reconfiguring mon.smithi105 (unknown last config time)... 2024-04-06T02:44:31.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:31 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:44:31.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:31 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:44:31.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:31 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:44:31.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:31 smithi184 ceph-mon[37439]: Reconfiguring daemon mon.smithi105 on smithi105 2024-04-06T02:44:31.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:31 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/1820937376' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:44:31.536 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:31 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:31.536 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:31 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:31.536 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:31 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:44:31.536 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:31 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:44:31.536 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:31 smithi105 ceph-mon[29228]: Updating smithi105:/etc/ceph/ceph.conf 2024-04-06T02:44:31.536 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:31 smithi105 ceph-mon[29228]: Updating smithi184:/etc/ceph/ceph.conf 2024-04-06T02:44:31.536 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:31 smithi105 ceph-mon[29228]: Updating smithi105:/var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:44:31.536 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:31 smithi105 ceph-mon[29228]: Updating smithi184:/var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T02:44:31.536 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:31 smithi105 ceph-mon[29228]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:31.537 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:31 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:31.537 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:31 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:31.537 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:31 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:31.537 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:31 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:31.537 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:31 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:31.537 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:31 smithi105 ceph-mon[29228]: Reconfiguring mon.smithi105 (unknown last config time)... 2024-04-06T02:44:31.537 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:31 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:44:31.537 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:31 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:44:31.537 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:31 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:44:31.537 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:31 smithi105 ceph-mon[29228]: Reconfiguring daemon mon.smithi105 on smithi105 2024-04-06T02:44:31.537 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:31 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/1820937376' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:44:32.803 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-06T02:44:32.803 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-06T02:44:32.803 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.conf 2024-04-06T02:44:32.832 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-06T02:44:32.832 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:44:32.899 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-04-06T02:44:32.899 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/ceph.conf 2024-04-06T02:44:32.934 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-04-06T02:44:32.935 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-06T02:44:33.006 INFO:tasks.cephadm:Deploying OSDs... 2024-04-06T02:44:33.007 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-06T02:44:33.007 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/scratch_devs of=/dev/stdout 2024-04-06T02:44:33.022 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-06T02:44:33.022 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_1 2024-04-06T02:44:33.076 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-06T02:44:33.076 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T02:44:33.076 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-06T02:44:33.076 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T02:44:33.076 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T02:44:33.077 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-04-06 02:43:46.212522684 +0000 2024-04-06T02:44:33.077 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-04-06 02:37:36.648669743 +0000 2024-04-06T02:44:33.077 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-04-06 02:37:36.648669743 +0000 2024-04-06T02:44:33.077 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-04-06 02:37:36.648669743 +0000 2024-04-06T02:44:33.077 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-06T02:44:33.142 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-04-06T02:44:33.142 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-04-06T02:44:33.142 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000209755 s, 2.4 MB/s 2024-04-06T02:44:33.143 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-06T02:44:33.199 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_2 2024-04-06T02:44:33.253 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-06T02:44:33.253 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T02:44:33.253 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-06T02:44:33.253 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T02:44:33.253 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T02:44:33.253 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-04-06 02:43:46.212522684 +0000 2024-04-06T02:44:33.253 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-04-06 02:37:37.009662553 +0000 2024-04-06T02:44:33.253 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-04-06 02:37:37.009662553 +0000 2024-04-06T02:44:33.253 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-04-06 02:37:37.009662553 +0000 2024-04-06T02:44:33.253 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-06T02:44:33.320 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-04-06T02:44:33.320 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-04-06T02:44:33.320 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000231598 s, 2.2 MB/s 2024-04-06T02:44:33.321 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-06T02:44:33.377 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_3 2024-04-06T02:44:33.431 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-06T02:44:33.431 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T02:44:33.431 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-06T02:44:33.431 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T02:44:33.431 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T02:44:33.431 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-04-06 02:43:46.212522684 +0000 2024-04-06T02:44:33.432 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-04-06 02:37:37.352655721 +0000 2024-04-06T02:44:33.432 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-04-06 02:37:37.352655721 +0000 2024-04-06T02:44:33.432 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-04-06 02:37:37.352655721 +0000 2024-04-06T02:44:33.432 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-06T02:44:33.498 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-04-06T02:44:33.498 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-04-06T02:44:33.498 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000228461 s, 2.2 MB/s 2024-04-06T02:44:33.499 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-06T02:44:33.555 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_4 2024-04-06T02:44:33.609 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-06T02:44:33.609 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T02:44:33.609 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-06T02:44:33.609 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T02:44:33.609 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T02:44:33.609 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-04-06 02:43:46.213522665 +0000 2024-04-06T02:44:33.609 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-04-06 02:37:37.707648649 +0000 2024-04-06T02:44:33.609 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-04-06 02:37:37.707648649 +0000 2024-04-06T02:44:33.610 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-04-06 02:37:37.707648649 +0000 2024-04-06T02:44:33.610 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-06T02:44:33.673 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:33 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:33.673 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:33 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:33.673 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:33 smithi105 ceph-mon[29228]: Reconfiguring mgr.smithi105.vqqcpx (unknown last config time)... 2024-04-06T02:44:33.673 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:33 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi105.vqqcpx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:44:33.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:33 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:44:33.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:33 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:44:33.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:33 smithi105 ceph-mon[29228]: Reconfiguring daemon mgr.smithi105.vqqcpx on smithi105 2024-04-06T02:44:33.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:33 smithi105 ceph-mon[29228]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:33.675 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-04-06T02:44:33.675 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-04-06T02:44:33.675 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000187839 s, 2.7 MB/s 2024-04-06T02:44:33.676 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-06T02:44:33.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:33 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:33.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:33 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:33.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:33 smithi184 ceph-mon[37439]: Reconfiguring mgr.smithi105.vqqcpx (unknown last config time)... 2024-04-06T02:44:33.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:33 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi105.vqqcpx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:44:33.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:33 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:44:33.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:33 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:44:33.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:33 smithi184 ceph-mon[37439]: Reconfiguring daemon mgr.smithi105.vqqcpx on smithi105 2024-04-06T02:44:33.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:33 smithi184 ceph-mon[37439]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:33.732 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-04-06T02:44:33.732 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/scratch_devs of=/dev/stdout 2024-04-06T02:44:33.751 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-06T02:44:33.751 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_1 2024-04-06T02:44:33.810 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-06T02:44:33.810 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T02:44:33.811 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-06T02:44:33.811 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T02:44:33.811 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T02:44:33.811 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-04-06 02:44:20.932958790 +0000 2024-04-06T02:44:33.811 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-04-06 02:37:36.654823389 +0000 2024-04-06T02:44:33.811 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-04-06 02:37:36.654823389 +0000 2024-04-06T02:44:33.811 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-04-06 02:37:36.654823389 +0000 2024-04-06T02:44:33.811 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-06T02:44:33.880 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-04-06T02:44:33.880 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-04-06T02:44:33.880 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000231736 s, 2.2 MB/s 2024-04-06T02:44:33.882 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-06T02:44:33.939 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_2 2024-04-06T02:44:33.995 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-06T02:44:33.996 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T02:44:33.996 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-06T02:44:33.996 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T02:44:33.996 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T02:44:33.996 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-04-06 02:44:20.932958790 +0000 2024-04-06T02:44:33.996 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-04-06 02:37:37.005816405 +0000 2024-04-06T02:44:33.996 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-04-06 02:37:37.005816405 +0000 2024-04-06T02:44:33.996 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-04-06 02:37:37.005816405 +0000 2024-04-06T02:44:33.996 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-06T02:44:34.063 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-04-06T02:44:34.063 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-04-06T02:44:34.063 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.00019619 s, 2.6 MB/s 2024-04-06T02:44:34.065 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-06T02:44:34.121 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_3 2024-04-06T02:44:34.177 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-06T02:44:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T02:44:34.178 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-04-06T02:44:34.178 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T02:44:34.178 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T02:44:34.178 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-04-06 02:44:20.932958790 +0000 2024-04-06T02:44:34.178 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-04-06 02:37:37.335809840 +0000 2024-04-06T02:44:34.178 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-04-06 02:37:37.335809840 +0000 2024-04-06T02:44:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-04-06 02:37:37.335809840 +0000 2024-04-06T02:44:34.178 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-06T02:44:34.244 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-04-06T02:44:34.245 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-04-06T02:44:34.245 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000185049 s, 2.8 MB/s 2024-04-06T02:44:34.246 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-06T02:44:34.302 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_4 2024-04-06T02:44:34.357 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-06T02:44:34.357 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T02:44:34.357 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-06T02:44:34.357 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T02:44:34.357 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-04-06T02:44:34.357 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-04-06 02:44:20.933958770 +0000 2024-04-06T02:44:34.357 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-04-06 02:37:37.672803135 +0000 2024-04-06T02:44:34.357 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-04-06 02:37:37.672803135 +0000 2024-04-06T02:44:34.357 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-04-06 02:37:37.672803135 +0000 2024-04-06T02:44:34.358 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-06T02:44:34.423 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-04-06T02:44:34.423 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-04-06T02:44:34.423 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000198452 s, 2.6 MB/s 2024-04-06T02:44:34.424 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-06T02:44:34.480 INFO:tasks.cephadm:Deploying osd.0 on smithi105 with /dev/vg_nvme/lv_4... 2024-04-06T02:44:34.480 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-06T02:44:35.161 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:44:35.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:35 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:35.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:35 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:35.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:35 smithi184 ceph-mon[37439]: Reconfiguring ceph-exporter.smithi105 (monmap changed)... 2024-04-06T02:44:35.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:35 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi105", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-06T02:44:35.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:35 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:44:35.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:35 smithi184 ceph-mon[37439]: Reconfiguring daemon ceph-exporter.smithi105 on smithi105 2024-04-06T02:44:35.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:35 smithi184 ceph-mon[37439]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:35.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:35 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:35.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:35 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:35.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:35 smithi105 ceph-mon[29228]: Reconfiguring ceph-exporter.smithi105 (monmap changed)... 2024-04-06T02:44:35.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:35 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi105", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-06T02:44:35.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:35 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:44:35.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:35 smithi105 ceph-mon[29228]: Reconfiguring daemon ceph-exporter.smithi105 on smithi105 2024-04-06T02:44:35.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:35 smithi105 ceph-mon[29228]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:36.788 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:36 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:36.788 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:36 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:36.788 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:36 smithi105 ceph-mon[29228]: Reconfiguring crash.smithi105 (monmap changed)... 2024-04-06T02:44:36.788 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:36 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi105", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T02:44:36.788 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:36 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:44:36.788 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:36 smithi105 ceph-mon[29228]: Reconfiguring daemon crash.smithi105 on smithi105 2024-04-06T02:44:36.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:36 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:36.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:36 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:36.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:36 smithi184 ceph-mon[37439]: Reconfiguring crash.smithi105 (monmap changed)... 2024-04-06T02:44:36.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:36 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi105", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T02:44:36.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:36 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:44:36.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:36 smithi184 ceph-mon[37439]: Reconfiguring daemon crash.smithi105 on smithi105 2024-04-06T02:44:37.519 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-06T02:44:37.519 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-06T02:44:37.520 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-04-06T02:44:37.520 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-04-06T02:44:37.520 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115191 s, 910 MB/s 2024-04-06T02:44:37.520 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-04-06T02:44:37.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:37 smithi105 ceph-mon[29228]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:37.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:37 smithi184 ceph-mon[37439]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:38.134 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi105:vg_nvme/lv_4 2024-04-06T02:44:38.878 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:44:39.996 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:39 smithi105 ceph-mon[29228]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:39.996 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:39 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:39.996 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:39 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:39.996 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:39 smithi105 ceph-mon[29228]: Reconfiguring alertmanager.smithi105 (dependencies changed)... 2024-04-06T02:44:39.996 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:39 smithi105 ceph-mon[29228]: Reconfiguring daemon alertmanager.smithi105 on smithi105 2024-04-06T02:44:39.996 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:39 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:44:40.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:39 smithi184 ceph-mon[37439]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:40.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:39 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:40.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:39 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:40.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:39 smithi184 ceph-mon[37439]: Reconfiguring alertmanager.smithi105 (dependencies changed)... 2024-04-06T02:44:40.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:39 smithi184 ceph-mon[37439]: Reconfiguring daemon alertmanager.smithi105 on smithi105 2024-04-06T02:44:40.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:39 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:44:41.866 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:41 smithi105 ceph-mon[29228]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:42.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:41 smithi184 ceph-mon[37439]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:43.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.657+0000 7f40530d2700 1 -- 172.21.15.105:0/3748993172 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f404c0ffab0 msgr2=0x7f404c0ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:44:43.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.657+0000 7f40530d2700 1 --2- 172.21.15.105:0/3748993172 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f404c0ffab0 0x7f404c0ffed0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4040009910 tx=0x7f4040009c20 comp rx=0 tx=0).stop 2024-04-06T02:44:43.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.657+0000 7f40530d2700 1 -- 172.21.15.105:0/3748993172 shutdown_connections 2024-04-06T02:44:43.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.657+0000 7f40530d2700 1 --2- 172.21.15.105:0/3748993172 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f404c100cb0 0x7f404c101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:43.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.657+0000 7f40530d2700 1 --2- 172.21.15.105:0/3748993172 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f404c0ffab0 0x7f404c0ffed0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:43.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.657+0000 7f40530d2700 1 -- 172.21.15.105:0/3748993172 >> 172.21.15.105:0/3748993172 conn(0x7f404c0fb050 msgr2=0x7f404c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:44:43.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.658+0000 7f40530d2700 1 -- 172.21.15.105:0/3748993172 shutdown_connections 2024-04-06T02:44:43.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.658+0000 7f40530d2700 1 -- 172.21.15.105:0/3748993172 wait complete. 2024-04-06T02:44:43.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.658+0000 7f40530d2700 1 Processor -- start 2024-04-06T02:44:43.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.658+0000 7f40530d2700 1 -- start start 2024-04-06T02:44:43.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.658+0000 7f40530d2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f404c0ffab0 0x7f404c1963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:44:43.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.658+0000 7f40530d2700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f404c100cb0 0x7f404c196900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:44:43.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.658+0000 7f40530d2700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404c196f20 con 0x7f404c0ffab0 2024-04-06T02:44:43.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.658+0000 7f40530d2700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404c197060 con 0x7f404c100cb0 2024-04-06T02:44:43.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.658+0000 7f4050e6e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f404c0ffab0 0x7f404c1963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:44:43.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.658+0000 7f4050e6e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f404c0ffab0 0x7f404c1963c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:48992/0 (socket says 172.21.15.105:48992) 2024-04-06T02:44:43.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.658+0000 7f4050e6e700 1 -- 172.21.15.105:0/3704889572 learned_addr learned my addr 172.21.15.105:0/3704889572 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:44:43.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.659+0000 7f4050e6e700 1 -- 172.21.15.105:0/3704889572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f404c100cb0 msgr2=0x7f404c196900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:44:43.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.659+0000 7f404bfff700 1 --2- 172.21.15.105:0/3704889572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f404c100cb0 0x7f404c196900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:44:43.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.659+0000 7f4050e6e700 1 --2- 172.21.15.105:0/3704889572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f404c100cb0 0x7f404c196900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:44:43.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.659+0000 7f4050e6e700 1 -- 172.21.15.105:0/3704889572 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40400095f0 con 0x7f404c0ffab0 2024-04-06T02:44:43.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.659+0000 7f4050e6e700 1 --2- 172.21.15.105:0/3704889572 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f404c0ffab0 0x7f404c1963c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f404000bda0 tx=0x7f404000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:44:43.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.659+0000 7f4049ffb700 1 -- 172.21.15.105:0/3704889572 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f404001d070 con 0x7f404c0ffab0 2024-04-06T02:44:43.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.659+0000 7f4049ffb700 1 -- 172.21.15.105:0/3704889572 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4040004c50 con 0x7f404c0ffab0 2024-04-06T02:44:43.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.659+0000 7f40530d2700 1 -- 172.21.15.105:0/3704889572 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f404c19bab0 con 0x7f404c0ffab0 2024-04-06T02:44:43.166 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.659+0000 7f4049ffb700 1 -- 172.21.15.105:0/3704889572 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f404000f690 con 0x7f404c0ffab0 2024-04-06T02:44:43.166 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.659+0000 7f40530d2700 1 -- 172.21.15.105:0/3704889572 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f404c19c020 con 0x7f404c0ffab0 2024-04-06T02:44:43.166 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.660+0000 7f40530d2700 1 -- 172.21.15.105:0/3704889572 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f404c108690 con 0x7f404c0ffab0 2024-04-06T02:44:43.166 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.661+0000 7f4049ffb700 1 -- 172.21.15.105:0/3704889572 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7f40400229b0 con 0x7f404c0ffab0 2024-04-06T02:44:43.166 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.661+0000 7f4049ffb700 1 --2- 172.21.15.105:0/3704889572 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f403c06c460 0x7f403c06e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:44:43.166 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.661+0000 7f4049ffb700 1 -- 172.21.15.105:0/3704889572 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f404008be70 con 0x7f404c0ffab0 2024-04-06T02:44:43.167 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.661+0000 7f404bfff700 1 --2- 172.21.15.105:0/3704889572 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f403c06c460 0x7f403c06e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:44:43.167 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.662+0000 7f404bfff700 1 --2- 172.21.15.105:0/3704889572 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f403c06c460 0x7f403c06e920 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4038006fd0 tx=0x7f4038008040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:44:43.167 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.666+0000 7f4049ffb700 1 -- 172.21.15.105:0/3704889572 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f404005b240 con 0x7f404c0ffab0 2024-04-06T02:44:43.167 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:44:42.866+0000 7f40530d2700 1 -- 172.21.15.105:0/3704889572 --> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f404c0693b0 con 0x7f403c06c460 2024-04-06T02:44:43.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:43 smithi105 ceph-mon[29228]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:43.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:43 smithi105 ceph-mon[29228]: from='client.14258 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:44:43.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:43 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:44:43.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:43 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:44:43.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:43 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:44:43.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:43 smithi184 ceph-mon[37439]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:43.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:43 smithi184 ceph-mon[37439]: from='client.14258 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:44:43.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:43 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:44:43.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:43 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:44:43.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:43 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:44:45.792 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:45 smithi105 ceph-mon[29228]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:45.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:45 smithi184 ceph-mon[37439]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:47.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:47 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:47.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:47 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:47.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:47 smithi105 ceph-mon[29228]: Reconfiguring grafana.smithi105 (dependencies changed)... 2024-04-06T02:44:47.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:47 smithi105 ceph-mon[29228]: Reconfiguring daemon grafana.smithi105 on smithi105 2024-04-06T02:44:47.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:47 smithi105 ceph-mon[29228]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:47.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:47 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:47.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:47 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:47.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:47 smithi184 ceph-mon[37439]: Reconfiguring grafana.smithi105 (dependencies changed)... 2024-04-06T02:44:47.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:47 smithi184 ceph-mon[37439]: Reconfiguring daemon grafana.smithi105 on smithi105 2024-04-06T02:44:47.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:47 smithi184 ceph-mon[37439]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:48.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:48 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/2682773727' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41c20c92-5314-4354-b659-cb449ef98c81"}]: dispatch 2024-04-06T02:44:48.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:48 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/2682773727' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "41c20c92-5314-4354-b659-cb449ef98c81"}]': finished 2024-04-06T02:44:48.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:48 smithi105 ceph-mon[29228]: osdmap e6: 1 total, 0 up, 1 in 2024-04-06T02:44:48.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:48 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:44:48.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:48 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/1562457975' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:44:48.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:48 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/2682773727' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41c20c92-5314-4354-b659-cb449ef98c81"}]: dispatch 2024-04-06T02:44:48.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:48 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/2682773727' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "41c20c92-5314-4354-b659-cb449ef98c81"}]': finished 2024-04-06T02:44:48.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:48 smithi184 ceph-mon[37439]: osdmap e6: 1 total, 0 up, 1 in 2024-04-06T02:44:48.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:48 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:44:48.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:48 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/1562457975' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:44:49.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:49 smithi105 ceph-mon[29228]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:49.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:49 smithi184 ceph-mon[37439]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:51.753 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:51 smithi105 ceph-mon[29228]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:51.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:51 smithi184 ceph-mon[37439]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:53.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:53 smithi105 ceph-mon[29228]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:53.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:53 smithi184 ceph-mon[37439]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:54.897 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:54 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:44:54.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:54 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:44:55.921 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:55 smithi105 ceph-mon[29228]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:55.930 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:55 smithi184 ceph-mon[37439]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:57.724 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:57 smithi105 ceph-mon[29228]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:57.724 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:57 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:57.724 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:57 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:57.724 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:57 smithi105 ceph-mon[29228]: Reconfiguring prometheus.smithi105 (dependencies changed)... 2024-04-06T02:44:57.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:57 smithi184 ceph-mon[37439]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:57.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:57 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:57.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:57 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:44:57.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:57 smithi184 ceph-mon[37439]: Reconfiguring prometheus.smithi105 (dependencies changed)... 2024-04-06T02:44:58.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:58 smithi105 ceph-mon[29228]: Reconfiguring daemon prometheus.smithi105 on smithi105 2024-04-06T02:44:58.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:58 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-06T02:44:58.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:58 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:44:58.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:58 smithi184 ceph-mon[37439]: Reconfiguring daemon prometheus.smithi105 on smithi105 2024-04-06T02:44:58.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:58 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-06T02:44:58.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:58 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:44:59.854 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:59 smithi105 ceph-mon[29228]: Deploying daemon osd.0 on smithi105 2024-04-06T02:44:59.854 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:44:59 smithi105 ceph-mon[29228]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:44:59.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:59 smithi184 ceph-mon[37439]: Deploying daemon osd.0 on smithi105 2024-04-06T02:44:59.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:44:59 smithi184 ceph-mon[37439]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:45:01.952 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:01 smithi105 ceph-mon[29228]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:45:01.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:01 smithi184 ceph-mon[37439]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:45:03.633 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:03 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:03.633 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:03 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:03.633 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:03 smithi105 ceph-mon[29228]: Reconfiguring ceph-exporter.smithi184 (monmap changed)... 2024-04-06T02:45:03.634 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:03 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi184", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-06T02:45:03.634 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:03 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:45:03.634 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:03 smithi105 ceph-mon[29228]: Reconfiguring daemon ceph-exporter.smithi184 on smithi184 2024-04-06T02:45:03.634 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:03 smithi105 ceph-mon[29228]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:45:03.641 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:03 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:03.641 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:03 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:03.641 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:03 smithi184 ceph-mon[37439]: Reconfiguring ceph-exporter.smithi184 (monmap changed)... 2024-04-06T02:45:03.641 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:03 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi184", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-06T02:45:03.641 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:03 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:45:03.641 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:03 smithi184 ceph-mon[37439]: Reconfiguring daemon ceph-exporter.smithi184 on smithi184 2024-04-06T02:45:03.641 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:03 smithi184 ceph-mon[37439]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:45:05.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:04 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:05.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:04 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:05.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:04 smithi184 ceph-mon[37439]: Reconfiguring crash.smithi184 (monmap changed)... 2024-04-06T02:45:05.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:04 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi184", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T02:45:05.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:04 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:45:05.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:04 smithi184 ceph-mon[37439]: Reconfiguring daemon crash.smithi184 on smithi184 2024-04-06T02:45:05.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:04 smithi184 ceph-mon[37439]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:45:05.325 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:04 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:05.325 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:04 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:05.325 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:04 smithi105 ceph-mon[29228]: Reconfiguring crash.smithi184 (monmap changed)... 2024-04-06T02:45:05.325 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:04 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi184", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-06T02:45:05.325 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:04 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:45:05.325 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:04 smithi105 ceph-mon[29228]: Reconfiguring daemon crash.smithi184 on smithi184 2024-04-06T02:45:05.325 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:04 smithi105 ceph-mon[29228]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:45:06.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:06 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:06.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:06 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:06.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:06 smithi184 ceph-mon[37439]: Reconfiguring mgr.smithi184.fzhhql (monmap changed)... 2024-04-06T02:45:06.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:06 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi184.fzhhql", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:45:06.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:06 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:45:06.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:06 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:45:06.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:06 smithi184 ceph-mon[37439]: Reconfiguring daemon mgr.smithi184.fzhhql on smithi184 2024-04-06T02:45:06.364 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:06 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:06.365 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:06 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:06.365 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:06 smithi105 ceph-mon[29228]: Reconfiguring mgr.smithi184.fzhhql (monmap changed)... 2024-04-06T02:45:06.365 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:06 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi184.fzhhql", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:45:06.365 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:06 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:45:06.365 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:06 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:45:06.365 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:06 smithi105 ceph-mon[29228]: Reconfiguring daemon mgr.smithi184.fzhhql on smithi184 2024-04-06T02:45:07.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:07 smithi105 ceph-mon[29228]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:45:07.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:07 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:07.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:07 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:07.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:07 smithi105 ceph-mon[29228]: Reconfiguring mon.smithi184 (monmap changed)... 2024-04-06T02:45:07.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:07 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:45:07.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:07 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:45:07.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:07 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:45:07.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:07 smithi105 ceph-mon[29228]: Reconfiguring daemon mon.smithi184 on smithi184 2024-04-06T02:45:07.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:07 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:07.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:07 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:07.952 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:07 smithi184 ceph-mon[37439]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:45:07.952 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:07 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:07.952 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:07 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:07.952 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:07 smithi184 ceph-mon[37439]: Reconfiguring mon.smithi184 (monmap changed)... 2024-04-06T02:45:07.952 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:07 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T02:45:07.952 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:07 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T02:45:07.953 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:07 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:45:07.953 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:07 smithi184 ceph-mon[37439]: Reconfiguring daemon mon.smithi184 on smithi184 2024-04-06T02:45:07.953 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:07 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:07.953 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:07 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:09.010 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:09 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:09.010 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:09 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:09.010 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:09 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T02:45:09.010 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:09 smithi105 ceph-mon[29228]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T02:45:09.010 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:09 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi105.front.sepia.ceph.com:9093"}]: dispatch 2024-04-06T02:45:09.010 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:09 smithi105 ceph-mon[29228]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi105.front.sepia.ceph.com:9093"}]: dispatch 2024-04-06T02:45:09.010 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:09 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:09.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:09 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T02:45:09.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:09 smithi105 ceph-mon[29228]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T02:45:09.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:09 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi105.front.sepia.ceph.com:3000"}]: dispatch 2024-04-06T02:45:09.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:09 smithi105 ceph-mon[29228]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi105.front.sepia.ceph.com:3000"}]: dispatch 2024-04-06T02:45:09.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:09 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:09.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:09 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T02:45:09.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:09 smithi105 ceph-mon[29228]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T02:45:09.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:09 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi105.front.sepia.ceph.com:9095"}]: dispatch 2024-04-06T02:45:09.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:09 smithi105 ceph-mon[29228]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi105.front.sepia.ceph.com:9095"}]: dispatch 2024-04-06T02:45:09.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:09 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:09.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:09 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:45:09.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:09 smithi105 ceph-mon[29228]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:45:09.307 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:09 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:09.307 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:09 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:09.307 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:09 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T02:45:09.307 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:09 smithi184 ceph-mon[37439]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T02:45:09.307 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:09 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi105.front.sepia.ceph.com:9093"}]: dispatch 2024-04-06T02:45:09.307 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:09 smithi184 ceph-mon[37439]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi105.front.sepia.ceph.com:9093"}]: dispatch 2024-04-06T02:45:09.307 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:09 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:09.307 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:09 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T02:45:09.307 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:09 smithi184 ceph-mon[37439]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T02:45:09.308 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:09 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi105.front.sepia.ceph.com:3000"}]: dispatch 2024-04-06T02:45:09.308 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:09 smithi184 ceph-mon[37439]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi105.front.sepia.ceph.com:3000"}]: dispatch 2024-04-06T02:45:09.308 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:09 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:09.308 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:09 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T02:45:09.308 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:09 smithi184 ceph-mon[37439]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T02:45:09.308 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:09 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi105.front.sepia.ceph.com:9095"}]: dispatch 2024-04-06T02:45:09.308 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:09 smithi184 ceph-mon[37439]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi105.front.sepia.ceph.com:9095"}]: dispatch 2024-04-06T02:45:09.308 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:09 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:09.308 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:09 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:45:09.308 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:09 smithi184 ceph-mon[37439]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:45:10.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:10 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:45:10.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:10 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:10.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:10 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:10.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:10 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:45:10.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:10 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:10.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:10 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:10.736 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 0 on host 'smithi105' 2024-04-06T02:45:10.736 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:10.750+0000 7f4049ffb700 1 -- 172.21.15.105:0/3704889572 <== mgr.14221 v2:172.21.15.105:6800/2334550011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f404c0693b0 con 0x7f403c06c460 2024-04-06T02:45:10.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:10.753+0000 7f40530d2700 1 -- 172.21.15.105:0/3704889572 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f403c06c460 msgr2=0x7f403c06e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:45:10.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:10.753+0000 7f40530d2700 1 --2- 172.21.15.105:0/3704889572 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f403c06c460 0x7f403c06e920 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4038006fd0 tx=0x7f4038008040 comp rx=0 tx=0).stop 2024-04-06T02:45:10.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:10.753+0000 7f40530d2700 1 -- 172.21.15.105:0/3704889572 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f404c0ffab0 msgr2=0x7f404c1963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:45:10.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:10.753+0000 7f40530d2700 1 --2- 172.21.15.105:0/3704889572 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f404c0ffab0 0x7f404c1963c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f404000bda0 tx=0x7f404000be80 comp rx=0 tx=0).stop 2024-04-06T02:45:10.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:10.753+0000 7f40530d2700 1 -- 172.21.15.105:0/3704889572 shutdown_connections 2024-04-06T02:45:10.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:10.753+0000 7f40530d2700 1 --2- 172.21.15.105:0/3704889572 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f403c06c460 0x7f403c06e920 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:45:10.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:10.753+0000 7f40530d2700 1 --2- 172.21.15.105:0/3704889572 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f404c0ffab0 0x7f404c1963c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:45:10.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:10.753+0000 7f40530d2700 1 --2- 172.21.15.105:0/3704889572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f404c100cb0 0x7f404c196900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:45:10.739 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:10.753+0000 7f40530d2700 1 -- 172.21.15.105:0/3704889572 >> 172.21.15.105:0/3704889572 conn(0x7f404c0fb050 msgr2=0x7f404c068ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:45:10.739 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:10.753+0000 7f40530d2700 1 -- 172.21.15.105:0/3704889572 shutdown_connections 2024-04-06T02:45:10.739 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:10.753+0000 7f40530d2700 1 -- 172.21.15.105:0/3704889572 wait complete. 2024-04-06T02:45:11.795 DEBUG:teuthology.orchestra.run.smithi105:osd.0> sudo journalctl -f -n 0 -u ceph-130df578-f3bf-11ee-b649-cb9ed24678a4@osd.0.service 2024-04-06T02:45:11.797 INFO:tasks.cephadm:Deploying osd.1 on smithi105 with /dev/vg_nvme/lv_3... 2024-04-06T02:45:11.797 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-06T02:45:11.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:11 smithi184 ceph-mon[37439]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:45:11.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:11 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:11.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:11 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:11.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:11 smithi184 ceph-mon[37439]: from='osd.0 [v2:172.21.15.105:6802/2018109658,v1:172.21.15.105:6803/2018109658]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-06T02:45:11.999 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:11 smithi105 ceph-mon[29228]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:45:11.999 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:11 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:11.999 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:11 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:11.999 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:11 smithi105 ceph-mon[29228]: from='osd.0 [v2:172.21.15.105:6802/2018109658,v1:172.21.15.105:6803/2018109658]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-06T02:45:12.684 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:45:13.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:13 smithi105 ceph-mon[29228]: from='osd.0 [v2:172.21.15.105:6802/2018109658,v1:172.21.15.105:6803/2018109658]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-06T02:45:13.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:13 smithi105 ceph-mon[29228]: osdmap e7: 1 total, 0 up, 1 in 2024-04-06T02:45:13.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:13 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:45:13.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:13 smithi105 ceph-mon[29228]: from='osd.0 [v2:172.21.15.105:6802/2018109658,v1:172.21.15.105:6803/2018109658]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-06T02:45:13.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:13 smithi105 ceph-mon[29228]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:45:13.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:45:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:45:13.085+0000 7f11c276c700 -1 osd.0 0 waiting for initial osdmap 2024-04-06T02:45:13.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:45:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:45:13.091+0000 7f11bed62700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T02:45:13.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:13 smithi184 ceph-mon[37439]: from='osd.0 [v2:172.21.15.105:6802/2018109658,v1:172.21.15.105:6803/2018109658]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-06T02:45:13.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:13 smithi184 ceph-mon[37439]: osdmap e7: 1 total, 0 up, 1 in 2024-04-06T02:45:13.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:13 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:45:13.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:13 smithi184 ceph-mon[37439]: from='osd.0 [v2:172.21.15.105:6802/2018109658,v1:172.21.15.105:6803/2018109658]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-06T02:45:13.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:13 smithi184 ceph-mon[37439]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:45:14.297 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:14 smithi105 ceph-mon[29228]: from='osd.0 [v2:172.21.15.105:6802/2018109658,v1:172.21.15.105:6803/2018109658]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-06T02:45:14.297 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:14 smithi105 ceph-mon[29228]: osdmap e8: 1 total, 0 up, 1 in 2024-04-06T02:45:14.297 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:14 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:45:14.297 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:14 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:45:14.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:14 smithi184 ceph-mon[37439]: from='osd.0 [v2:172.21.15.105:6802/2018109658,v1:172.21.15.105:6803/2018109658]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-06T02:45:14.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:14 smithi184 ceph-mon[37439]: osdmap e8: 1 total, 0 up, 1 in 2024-04-06T02:45:14.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:14 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:45:14.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:14 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:45:15.362 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:15 smithi105 ceph-mon[29228]: purged_snaps scrub starts 2024-04-06T02:45:15.362 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:15 smithi105 ceph-mon[29228]: purged_snaps scrub ok 2024-04-06T02:45:15.362 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:15 smithi105 ceph-mon[29228]: osd.0 [v2:172.21.15.105:6802/2018109658,v1:172.21.15.105:6803/2018109658] boot 2024-04-06T02:45:15.362 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:15 smithi105 ceph-mon[29228]: osdmap e9: 1 total, 1 up, 1 in 2024-04-06T02:45:15.362 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:15 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:45:15.362 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:15 smithi105 ceph-mon[29228]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:45:15.381 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:15 smithi184 ceph-mon[37439]: purged_snaps scrub starts 2024-04-06T02:45:15.382 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:15 smithi184 ceph-mon[37439]: purged_snaps scrub ok 2024-04-06T02:45:15.382 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:15 smithi184 ceph-mon[37439]: osd.0 [v2:172.21.15.105:6802/2018109658,v1:172.21.15.105:6803/2018109658] boot 2024-04-06T02:45:15.382 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:15 smithi184 ceph-mon[37439]: osdmap e9: 1 total, 1 up, 1 in 2024-04-06T02:45:15.382 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:15 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T02:45:15.382 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:15 smithi184 ceph-mon[37439]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T02:45:16.343 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-06T02:45:16.343 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-06T02:45:16.343 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-04-06T02:45:16.343 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-04-06T02:45:16.343 INFO:teuthology.orchestra.run.smithi105.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0126604 s, 828 MB/s 2024-04-06T02:45:16.343 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 2024-04-06T02:45:16.343 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-04-06T02:45:16.344 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:16 smithi105 ceph-mon[29228]: osdmap e10: 1 total, 1 up, 1 in 2024-04-06T02:45:16.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:16 smithi184 ceph-mon[37439]: osdmap e10: 1 total, 1 up, 1 in 2024-04-06T02:45:17.041 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi105:vg_nvme/lv_3 2024-04-06T02:45:17.262 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:45:17.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:17 smithi105 ceph-mon[29228]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:45:17.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:17 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:17.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:17 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:17.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:17 smithi184 ceph-mon[37439]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:45:17.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:17 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:17.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:17 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:19.544 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.557+0000 7fe668372700 1 -- 172.21.15.105:0/3115099741 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe660103fa0 msgr2=0x7fe660106390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:45:19.544 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.557+0000 7fe668372700 1 --2- 172.21.15.105:0/3115099741 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe660103fa0 0x7fe660106390 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe650009910 tx=0x7fe650009c20 comp rx=0 tx=0).stop 2024-04-06T02:45:19.544 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.557+0000 7fe668372700 1 -- 172.21.15.105:0/3115099741 shutdown_connections 2024-04-06T02:45:19.544 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.557+0000 7fe668372700 1 --2- 172.21.15.105:0/3115099741 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe660103fa0 0x7fe660106390 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:45:19.544 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.557+0000 7fe668372700 1 --2- 172.21.15.105:0/3115099741 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe660101670 0x7fe660103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:45:19.544 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.557+0000 7fe668372700 1 -- 172.21.15.105:0/3115099741 >> 172.21.15.105:0/3115099741 conn(0x7fe6600fb080 msgr2=0x7fe6600fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:45:19.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.557+0000 7fe668372700 1 -- 172.21.15.105:0/3115099741 shutdown_connections 2024-04-06T02:45:19.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.557+0000 7fe668372700 1 -- 172.21.15.105:0/3115099741 wait complete. 2024-04-06T02:45:19.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.558+0000 7fe668372700 1 Processor -- start 2024-04-06T02:45:19.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.558+0000 7fe668372700 1 -- start start 2024-04-06T02:45:19.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.558+0000 7fe668372700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe660101670 0x7fe660196340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:45:19.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.558+0000 7fe668372700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe660103fa0 0x7fe660196880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:45:19.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.558+0000 7fe668372700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe660196ea0 con 0x7fe660103fa0 2024-04-06T02:45:19.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.558+0000 7fe668372700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe660196fe0 con 0x7fe660101670 2024-04-06T02:45:19.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.558+0000 7fe66590d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe660103fa0 0x7fe660196880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:45:19.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.558+0000 7fe66590d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe660103fa0 0x7fe660196880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50196/0 (socket says 172.21.15.105:50196) 2024-04-06T02:45:19.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.558+0000 7fe66590d700 1 -- 172.21.15.105:0/908658280 learned_addr learned my addr 172.21.15.105:0/908658280 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:45:19.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.558+0000 7fe66610e700 1 --2- 172.21.15.105:0/908658280 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe660101670 0x7fe660196340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:45:19.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.558+0000 7fe66590d700 1 -- 172.21.15.105:0/908658280 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe660101670 msgr2=0x7fe660196340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:45:19.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.558+0000 7fe66590d700 1 --2- 172.21.15.105:0/908658280 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe660101670 0x7fe660196340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:45:19.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.559+0000 7fe66590d700 1 -- 172.21.15.105:0/908658280 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6500095f0 con 0x7fe660103fa0 2024-04-06T02:45:19.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.559+0000 7fe66590d700 1 --2- 172.21.15.105:0/908658280 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe660103fa0 0x7fe660196880 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe650006010 tx=0x7fe65000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:45:19.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.559+0000 7fe6577fe700 1 -- 172.21.15.105:0/908658280 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe65001d070 con 0x7fe660103fa0 2024-04-06T02:45:19.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.559+0000 7fe6577fe700 1 -- 172.21.15.105:0/908658280 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe65000f460 con 0x7fe660103fa0 2024-04-06T02:45:19.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.559+0000 7fe668372700 1 -- 172.21.15.105:0/908658280 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe66019ba30 con 0x7fe660103fa0 2024-04-06T02:45:19.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.559+0000 7fe6577fe700 1 -- 172.21.15.105:0/908658280 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe650021640 con 0x7fe660103fa0 2024-04-06T02:45:19.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.559+0000 7fe668372700 1 -- 172.21.15.105:0/908658280 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe66019bf20 con 0x7fe660103fa0 2024-04-06T02:45:19.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.560+0000 7fe668372700 1 -- 172.21.15.105:0/908658280 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe660190550 con 0x7fe660103fa0 2024-04-06T02:45:19.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.560+0000 7fe6577fe700 1 -- 172.21.15.105:0/908658280 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7fe65000fab0 con 0x7fe660103fa0 2024-04-06T02:45:19.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.561+0000 7fe6577fe700 1 --2- 172.21.15.105:0/908658280 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fe64c06c460 0x7fe64c06e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:45:19.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.561+0000 7fe66610e700 1 --2- 172.21.15.105:0/908658280 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fe64c06c460 0x7fe64c06e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:45:19.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.561+0000 7fe6577fe700 1 -- 172.21.15.105:0/908658280 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1982+0+0 (secure 0 0 0) 0x7fe65008bd80 con 0x7fe660103fa0 2024-04-06T02:45:19.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.561+0000 7fe66610e700 1 --2- 172.21.15.105:0/908658280 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fe64c06c460 0x7fe64c06e920 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe65c005950 tx=0x7fe65c005cf0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:45:19.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.565+0000 7fe6577fe700 1 -- 172.21.15.105:0/908658280 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe65005b010 con 0x7fe660103fa0 2024-04-06T02:45:19.753 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:19.768+0000 7fe668372700 1 -- 172.21.15.105:0/908658280 --> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6600611d0 con 0x7fe64c06c460 2024-04-06T02:45:19.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:19 smithi105 ceph-mon[29228]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:45:19.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:19 smithi184 ceph-mon[37439]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:45:20.916 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:20 smithi105 ceph-mon[29228]: from='client.14276 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:45:20.917 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:20 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:45:20.917 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:20 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:45:20.917 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:20 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:45:20.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:20 smithi184 ceph-mon[37439]: from='client.14276 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:45:20.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:20 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:45:20.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:20 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:45:20.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:20 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:45:21.881 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:21 smithi105 ceph-mon[29228]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:45:21.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:21 smithi184 ceph-mon[37439]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:45:22.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:22 smithi184 ceph-mon[37439]: Detected new or changed devices on smithi105 2024-04-06T02:45:22.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:22 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:22.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:22 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:22.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:22 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:45:22.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:22 smithi184 ceph-mon[37439]: Adjusting osd_memory_target on smithi105 to 11840M 2024-04-06T02:45:22.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:22 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:22.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:22 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:45:22.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:22 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:45:22.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:22 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:22.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:22 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:45:23.153 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:22 smithi105 ceph-mon[29228]: Detected new or changed devices on smithi105 2024-04-06T02:45:23.153 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:22 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:23.153 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:22 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:23.153 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:22 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:45:23.153 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:22 smithi105 ceph-mon[29228]: Adjusting osd_memory_target on smithi105 to 11840M 2024-04-06T02:45:23.153 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:22 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:23.153 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:22 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:45:23.153 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:22 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:45:23.153 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:22 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:23.153 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:22 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:45:24.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:23 smithi105 ceph-mon[29228]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:45:24.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:23 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/682248670' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea3f1b19-62f8-488e-b62a-2340aef1112a"}]: dispatch 2024-04-06T02:45:24.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:23 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/682248670' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea3f1b19-62f8-488e-b62a-2340aef1112a"}]': finished 2024-04-06T02:45:24.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:23 smithi105 ceph-mon[29228]: osdmap e11: 2 total, 1 up, 2 in 2024-04-06T02:45:24.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:23 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:45:24.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:23 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:24.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:23 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:24.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:23 smithi184 ceph-mon[37439]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:45:24.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:23 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/682248670' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea3f1b19-62f8-488e-b62a-2340aef1112a"}]: dispatch 2024-04-06T02:45:24.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:23 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/682248670' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea3f1b19-62f8-488e-b62a-2340aef1112a"}]': finished 2024-04-06T02:45:24.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:23 smithi184 ceph-mon[37439]: osdmap e11: 2 total, 1 up, 2 in 2024-04-06T02:45:24.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:23 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:45:24.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:23 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:24.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:23 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:25.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:24 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/1408019939' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:45:25.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:24 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:45:25.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:24 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/1408019939' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:45:25.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:24 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:45:26.145 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:25 smithi105 ceph-mon[29228]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:45:26.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:25 smithi184 ceph-mon[37439]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:45:27.601 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:27.602 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:27.602 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:45:27.602 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:45:27.602 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:27.602 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:27 smithi105 ceph-mon[29228]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:45:27.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:27.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:27.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:45:27.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:45:27.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:27.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:27 smithi184 ceph-mon[37439]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:45:29.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:29 smithi105 ceph-mon[29228]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:45:29.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:29 smithi184 ceph-mon[37439]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:45:31.781 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:31 smithi105 ceph-mon[29228]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:45:31.781 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:31 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-06T02:45:31.781 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:31 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:45:31.781 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:31 smithi105 ceph-mon[29228]: Deploying daemon osd.1 on smithi105 2024-04-06T02:45:31.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:31 smithi184 ceph-mon[37439]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:45:31.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:31 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-06T02:45:31.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:31 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:45:31.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:31 smithi184 ceph-mon[37439]: Deploying daemon osd.1 on smithi105 2024-04-06T02:45:33.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:33 smithi105 ceph-mon[29228]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:45:33.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:33 smithi184 ceph-mon[37439]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:45:35.886 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:35 smithi105 ceph-mon[29228]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:45:35.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:35 smithi184 ceph-mon[37439]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:45:37.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:37 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:37.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:37 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:37.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:37 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:45:37.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:37 smithi105 ceph-mon[29228]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:45:37.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:37 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:37.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:37 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:37.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:37 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:45:37.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:37 smithi184 ceph-mon[37439]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:45:39.404 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:39 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:39.404 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:39 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:39.404 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:39 smithi105 ceph-mon[29228]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:45:39.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:39.470+0000 7fe6577fe700 1 -- 172.21.15.105:0/908658280 <== mgr.14221 v2:172.21.15.105:6800/2334550011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe6600611d0 con 0x7fe64c06c460 2024-04-06T02:45:39.455 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 1 on host 'smithi105' 2024-04-06T02:45:39.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:39.472+0000 7fe668372700 1 -- 172.21.15.105:0/908658280 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fe64c06c460 msgr2=0x7fe64c06e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:45:39.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:39.472+0000 7fe668372700 1 --2- 172.21.15.105:0/908658280 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fe64c06c460 0x7fe64c06e920 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe65c005950 tx=0x7fe65c005cf0 comp rx=0 tx=0).stop 2024-04-06T02:45:39.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:39.472+0000 7fe668372700 1 -- 172.21.15.105:0/908658280 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe660103fa0 msgr2=0x7fe660196880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:45:39.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:39.472+0000 7fe668372700 1 --2- 172.21.15.105:0/908658280 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe660103fa0 0x7fe660196880 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe650006010 tx=0x7fe65000be20 comp rx=0 tx=0).stop 2024-04-06T02:45:39.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:39.472+0000 7fe668372700 1 -- 172.21.15.105:0/908658280 shutdown_connections 2024-04-06T02:45:39.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:39.472+0000 7fe668372700 1 --2- 172.21.15.105:0/908658280 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fe64c06c460 0x7fe64c06e920 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:45:39.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:39.472+0000 7fe668372700 1 --2- 172.21.15.105:0/908658280 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe660101670 0x7fe660196340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:45:39.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:39.472+0000 7fe668372700 1 --2- 172.21.15.105:0/908658280 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe660103fa0 0x7fe660196880 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:45:39.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:39.472+0000 7fe668372700 1 -- 172.21.15.105:0/908658280 >> 172.21.15.105:0/908658280 conn(0x7fe6600fb080 msgr2=0x7fe6600fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:45:39.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:39.472+0000 7fe668372700 1 -- 172.21.15.105:0/908658280 shutdown_connections 2024-04-06T02:45:39.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:39.472+0000 7fe668372700 1 -- 172.21.15.105:0/908658280 wait complete. 2024-04-06T02:45:39.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:39 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:39.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:39 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:39.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:39 smithi184 ceph-mon[37439]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:45:40.084 DEBUG:teuthology.orchestra.run.smithi105:osd.1> sudo journalctl -f -n 0 -u ceph-130df578-f3bf-11ee-b649-cb9ed24678a4@osd.1.service 2024-04-06T02:45:40.086 INFO:tasks.cephadm:Deploying osd.2 on smithi105 with /dev/vg_nvme/lv_2... 2024-04-06T02:45:40.086 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-06T02:45:40.720 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:40 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:40.720 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:40 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:40.720 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:40 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:45:40.720 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:40 smithi105 ceph-mon[29228]: from='osd.1 [v2:172.21.15.105:6810/1345191629,v1:172.21.15.105:6811/1345191629]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-06T02:45:40.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:40 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:40.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:40 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:40.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:40 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:45:40.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:40 smithi184 ceph-mon[37439]: from='osd.1 [v2:172.21.15.105:6810/1345191629,v1:172.21.15.105:6811/1345191629]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-06T02:45:40.990 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:45:41.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:41 smithi184 ceph-mon[37439]: from='osd.1 [v2:172.21.15.105:6810/1345191629,v1:172.21.15.105:6811/1345191629]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-06T02:45:41.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:41 smithi184 ceph-mon[37439]: osdmap e12: 2 total, 1 up, 2 in 2024-04-06T02:45:41.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:41 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:45:41.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:41 smithi184 ceph-mon[37439]: from='osd.1 [v2:172.21.15.105:6810/1345191629,v1:172.21.15.105:6811/1345191629]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-06T02:45:41.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:41 smithi184 ceph-mon[37439]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:45:41.742 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:41 smithi105 ceph-mon[29228]: from='osd.1 [v2:172.21.15.105:6810/1345191629,v1:172.21.15.105:6811/1345191629]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-06T02:45:41.742 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:41 smithi105 ceph-mon[29228]: osdmap e12: 2 total, 1 up, 2 in 2024-04-06T02:45:41.742 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:41 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:45:41.742 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:41 smithi105 ceph-mon[29228]: from='osd.1 [v2:172.21.15.105:6810/1345191629,v1:172.21.15.105:6811/1345191629]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-06T02:45:41.742 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:41 smithi105 ceph-mon[29228]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-06T02:45:41.742 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:45:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:45:41.463+0000 7fdfe7bba700 -1 osd.1 0 waiting for initial osdmap 2024-04-06T02:45:41.743 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:45:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:45:41.467+0000 7fdfde9a5700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T02:45:42.664 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:42 smithi105 ceph-mon[29228]: from='osd.1 [v2:172.21.15.105:6810/1345191629,v1:172.21.15.105:6811/1345191629]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-06T02:45:42.665 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:42 smithi105 ceph-mon[29228]: osdmap e13: 2 total, 1 up, 2 in 2024-04-06T02:45:42.665 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:42 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:45:42.665 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:42 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:45:42.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:42 smithi184 ceph-mon[37439]: from='osd.1 [v2:172.21.15.105:6810/1345191629,v1:172.21.15.105:6811/1345191629]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-06T02:45:42.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:42 smithi184 ceph-mon[37439]: osdmap e13: 2 total, 1 up, 2 in 2024-04-06T02:45:42.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:42 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:45:42.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:42 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:45:43.602 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:43 smithi105 ceph-mon[29228]: purged_snaps scrub starts 2024-04-06T02:45:43.602 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:43 smithi105 ceph-mon[29228]: purged_snaps scrub ok 2024-04-06T02:45:43.602 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:43 smithi105 ceph-mon[29228]: osd.1 [v2:172.21.15.105:6810/1345191629,v1:172.21.15.105:6811/1345191629] boot 2024-04-06T02:45:43.602 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:43 smithi105 ceph-mon[29228]: osdmap e14: 2 total, 2 up, 2 in 2024-04-06T02:45:43.602 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:43 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:45:43.602 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:43 smithi105 ceph-mon[29228]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:45:43.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:43 smithi184 ceph-mon[37439]: purged_snaps scrub starts 2024-04-06T02:45:43.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:43 smithi184 ceph-mon[37439]: purged_snaps scrub ok 2024-04-06T02:45:43.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:43 smithi184 ceph-mon[37439]: osd.1 [v2:172.21.15.105:6810/1345191629,v1:172.21.15.105:6811/1345191629] boot 2024-04-06T02:45:43.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:43 smithi184 ceph-mon[37439]: osdmap e14: 2 total, 2 up, 2 in 2024-04-06T02:45:43.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:43 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T02:45:43.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:43 smithi184 ceph-mon[37439]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:45:44.667 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:44 smithi105 ceph-mon[29228]: osdmap e15: 2 total, 2 up, 2 in 2024-04-06T02:45:44.667 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:44 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:44.667 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:44 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:44.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:44 smithi184 ceph-mon[37439]: osdmap e15: 2 total, 2 up, 2 in 2024-04-06T02:45:44.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:44 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:44.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:44 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:45.096 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-06T02:45:45.096 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-06T02:45:45.096 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-04-06T02:45:45.096 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-04-06T02:45:45.097 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126209 s, 831 MB/s 2024-04-06T02:45:45.097 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-04-06T02:45:45.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:45 smithi105 ceph-mon[29228]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:45:45.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:45 smithi184 ceph-mon[37439]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:45:46.132 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi105:vg_nvme/lv_2 2024-04-06T02:45:46.573 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:45:47.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:47 smithi105 ceph-mon[29228]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:45:47.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:47 smithi184 ceph-mon[37439]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:45:48.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.837+0000 7f68d1524700 1 -- 172.21.15.105:0/1396590853 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68cc101dc0 msgr2=0x7f68cc102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:45:48.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.837+0000 7f68d1524700 1 --2- 172.21.15.105:0/1396590853 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68cc101dc0 0x7f68cc102220 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f68c0009910 tx=0x7f68c0009c20 comp rx=0 tx=0).stop 2024-04-06T02:45:48.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.837+0000 7f68d1524700 1 -- 172.21.15.105:0/1396590853 shutdown_connections 2024-04-06T02:45:48.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.837+0000 7f68d1524700 1 --2- 172.21.15.105:0/1396590853 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68cc101dc0 0x7f68cc102220 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:45:48.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.837+0000 7f68d1524700 1 --2- 172.21.15.105:0/1396590853 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68cc100bc0 0x7f68cc100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:45:48.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.837+0000 7f68d1524700 1 -- 172.21.15.105:0/1396590853 >> 172.21.15.105:0/1396590853 conn(0x7f68cc0fc140 msgr2=0x7f68cc0fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:45:48.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.837+0000 7f68d1524700 1 -- 172.21.15.105:0/1396590853 shutdown_connections 2024-04-06T02:45:48.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.838+0000 7f68d1524700 1 -- 172.21.15.105:0/1396590853 wait complete. 2024-04-06T02:45:48.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.838+0000 7f68d1524700 1 Processor -- start 2024-04-06T02:45:48.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.838+0000 7f68d1524700 1 -- start start 2024-04-06T02:45:48.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.838+0000 7f68d1524700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68cc100bc0 0x7f68cc196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:45:48.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.838+0000 7f68d1524700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68cc101dc0 0x7f68cc1968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:45:48.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.838+0000 7f68d1524700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68cc196ed0 con 0x7f68cc100bc0 2024-04-06T02:45:48.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.838+0000 7f68d1524700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68cc197010 con 0x7f68cc101dc0 2024-04-06T02:45:48.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.838+0000 7f68caffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68cc100bc0 0x7f68cc196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:45:48.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.838+0000 7f68caffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68cc100bc0 0x7f68cc196370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38830/0 (socket says 172.21.15.105:38830) 2024-04-06T02:45:48.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.839+0000 7f68caffd700 1 -- 172.21.15.105:0/4127612518 learned_addr learned my addr 172.21.15.105:0/4127612518 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:45:48.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.839+0000 7f68ca7fc700 1 --2- 172.21.15.105:0/4127612518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68cc101dc0 0x7f68cc1968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:45:48.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.839+0000 7f68caffd700 1 -- 172.21.15.105:0/4127612518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68cc101dc0 msgr2=0x7f68cc1968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:45:48.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.839+0000 7f68caffd700 1 --2- 172.21.15.105:0/4127612518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68cc101dc0 0x7f68cc1968b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:45:48.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.839+0000 7f68caffd700 1 -- 172.21.15.105:0/4127612518 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c00095f0 con 0x7f68cc100bc0 2024-04-06T02:45:48.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.839+0000 7f68caffd700 1 --2- 172.21.15.105:0/4127612518 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68cc100bc0 0x7f68cc196370 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f68bc00d900 tx=0x7f68bc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:45:48.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.839+0000 7f68b3fff700 1 -- 172.21.15.105:0/4127612518 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68bc0041f0 con 0x7f68cc100bc0 2024-04-06T02:45:48.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.839+0000 7f68d1524700 1 -- 172.21.15.105:0/4127612518 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68cc19bac0 con 0x7f68cc100bc0 2024-04-06T02:45:48.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.839+0000 7f68b3fff700 1 -- 172.21.15.105:0/4127612518 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f68bc004350 con 0x7f68cc100bc0 2024-04-06T02:45:48.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.839+0000 7f68b3fff700 1 -- 172.21.15.105:0/4127612518 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68bc003da0 con 0x7f68cc100bc0 2024-04-06T02:45:48.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.839+0000 7f68d1524700 1 -- 172.21.15.105:0/4127612518 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68cc19c010 con 0x7f68cc100bc0 2024-04-06T02:45:48.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.840+0000 7f68d1524700 1 -- 172.21.15.105:0/4127612518 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68cc109720 con 0x7f68cc100bc0 2024-04-06T02:45:48.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.841+0000 7f68b3fff700 1 -- 172.21.15.105:0/4127612518 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7f68bc009730 con 0x7f68cc100bc0 2024-04-06T02:45:48.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.841+0000 7f68b3fff700 1 --2- 172.21.15.105:0/4127612518 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f68b406c460 0x7f68b406e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:45:48.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.841+0000 7f68b3fff700 1 -- 172.21.15.105:0/4127612518 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2414+0+0 (secure 0 0 0) 0x7f68bc08ac90 con 0x7f68cc100bc0 2024-04-06T02:45:48.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.841+0000 7f68ca7fc700 1 --2- 172.21.15.105:0/4127612518 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f68b406c460 0x7f68b406e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:45:48.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.841+0000 7f68ca7fc700 1 --2- 172.21.15.105:0/4127612518 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f68b406c460 0x7f68b406e920 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f68c000b5c0 tx=0x7f68c0005960 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:45:48.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:48.845+0000 7f68b3fff700 1 -- 172.21.15.105:0/4127612518 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f68bc059bd0 con 0x7f68cc100bc0 2024-04-06T02:45:49.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:45:49.048+0000 7f68d1524700 1 -- 172.21.15.105:0/4127612518 --> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f68cc106710 con 0x7f68b406c460 2024-04-06T02:45:49.740 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:49 smithi105 ceph-mon[29228]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:45:49.740 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:49 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:45:49.740 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:49 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:45:49.740 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:49 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:45:49.740 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:49 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:49.740 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:49 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:49.741 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:49 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:45:49.741 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:49 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:45:49.741 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:49 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:49.741 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:49 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:45:49.741 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:49 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:45:49.741 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:49 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:49.741 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:49 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:45:49.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:49 smithi184 ceph-mon[37439]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:45:49.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:49 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:45:49.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:49 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:45:49.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:49 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:45:49.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:49 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:49.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:49 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:49.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:49 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:45:49.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:49 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:45:49.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:49 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:49.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:49 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:45:49.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:49 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:45:49.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:49 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:49.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:49 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:45:50.755 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:50 smithi105 ceph-mon[29228]: from='client.14294 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:45:50.755 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:50 smithi105 ceph-mon[29228]: Detected new or changed devices on smithi105 2024-04-06T02:45:50.755 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:50 smithi105 ceph-mon[29228]: Adjusting osd_memory_target on smithi105 to 5920M 2024-04-06T02:45:50.940 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:50 smithi184 ceph-mon[37439]: from='client.14294 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:45:50.940 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:50 smithi184 ceph-mon[37439]: Detected new or changed devices on smithi105 2024-04-06T02:45:50.940 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:50 smithi184 ceph-mon[37439]: Adjusting osd_memory_target on smithi105 to 5920M 2024-04-06T02:45:51.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:51 smithi105 ceph-mon[29228]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:45:51.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:51 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:51.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:51 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:51.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:51 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/2779327506' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "def7693a-bb30-4c74-ae20-59a7a86ef9db"}]: dispatch 2024-04-06T02:45:51.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:51 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/2779327506' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "def7693a-bb30-4c74-ae20-59a7a86ef9db"}]': finished 2024-04-06T02:45:51.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:51 smithi105 ceph-mon[29228]: osdmap e16: 3 total, 2 up, 3 in 2024-04-06T02:45:51.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:51 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:45:51.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:51 smithi184 ceph-mon[37439]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:45:51.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:51 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:51.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:51 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:51.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:51 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/2779327506' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "def7693a-bb30-4c74-ae20-59a7a86ef9db"}]: dispatch 2024-04-06T02:45:51.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:51 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/2779327506' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "def7693a-bb30-4c74-ae20-59a7a86ef9db"}]': finished 2024-04-06T02:45:51.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:51 smithi184 ceph-mon[37439]: osdmap e16: 3 total, 2 up, 3 in 2024-04-06T02:45:51.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:51 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:45:52.917 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:52 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/3860734172' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:45:52.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:52 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/3860734172' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:45:53.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:53 smithi105 ceph-mon[29228]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:45:53.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:53 smithi184 ceph-mon[37439]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:45:54.758 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:54 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:45:54.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:54 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:45:55.820 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:55 smithi105 ceph-mon[29228]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:45:55.820 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:55 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:55.821 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:55 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:55.821 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:55 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:45:55.821 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:55 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:45:55.821 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:55 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:55.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:55 smithi184 ceph-mon[37439]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:45:55.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:55 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:55.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:55 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:55.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:55 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:45:55.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:55 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:45:55.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:55 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:45:57.860 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:57 smithi105 ceph-mon[29228]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:45:57.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:57 smithi184 ceph-mon[37439]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:45:59.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:59 smithi105 ceph-mon[29228]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:45:59.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:59 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-06T02:45:59.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:45:59 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:45:59.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:59 smithi184 ceph-mon[37439]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:45:59.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:59 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-06T02:45:59.982 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:45:59 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:46:00.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:00 smithi105 ceph-mon[29228]: Deploying daemon osd.2 on smithi105 2024-04-06T02:46:00.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:00 smithi184 ceph-mon[37439]: Deploying daemon osd.2 on smithi105 2024-04-06T02:46:01.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:01 smithi105 ceph-mon[29228]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:46:01.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:01 smithi184 ceph-mon[37439]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:46:03.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:03 smithi105 ceph-mon[29228]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:46:03.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:03 smithi184 ceph-mon[37439]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:46:05.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:05 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:05.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:05 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:05.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:05 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:46:05.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:05 smithi105 ceph-mon[29228]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:46:05.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:05 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:05.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:05 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:05.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:05 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:46:05.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:05 smithi184 ceph-mon[37439]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:46:07.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:07 smithi105 ceph-mon[29228]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:46:07.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:07 smithi184 ceph-mon[37439]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:46:08.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:46:08.070+0000 7f68b3fff700 1 -- 172.21.15.105:0/4127612518 <== mgr.14221 v2:172.21.15.105:6800/2334550011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f68cc106710 con 0x7f68b406c460 2024-04-06T02:46:08.056 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 2 on host 'smithi105' 2024-04-06T02:46:08.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:46:08.072+0000 7f68d1524700 1 -- 172.21.15.105:0/4127612518 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f68b406c460 msgr2=0x7f68b406e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:46:08.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:46:08.072+0000 7f68d1524700 1 --2- 172.21.15.105:0/4127612518 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f68b406c460 0x7f68b406e920 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f68c000b5c0 tx=0x7f68c0005960 comp rx=0 tx=0).stop 2024-04-06T02:46:08.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:46:08.072+0000 7f68d1524700 1 -- 172.21.15.105:0/4127612518 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68cc100bc0 msgr2=0x7f68cc196370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:46:08.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:46:08.072+0000 7f68d1524700 1 --2- 172.21.15.105:0/4127612518 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68cc100bc0 0x7f68cc196370 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f68bc00d900 tx=0x7f68bc00dcc0 comp rx=0 tx=0).stop 2024-04-06T02:46:08.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:46:08.073+0000 7f68d1524700 1 -- 172.21.15.105:0/4127612518 shutdown_connections 2024-04-06T02:46:08.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:46:08.073+0000 7f68d1524700 1 --2- 172.21.15.105:0/4127612518 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f68b406c460 0x7f68b406e920 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:46:08.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:46:08.073+0000 7f68d1524700 1 --2- 172.21.15.105:0/4127612518 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68cc100bc0 0x7f68cc196370 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:46:08.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:46:08.073+0000 7f68d1524700 1 --2- 172.21.15.105:0/4127612518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68cc101dc0 0x7f68cc1968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:46:08.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:46:08.073+0000 7f68d1524700 1 -- 172.21.15.105:0/4127612518 >> 172.21.15.105:0/4127612518 conn(0x7f68cc0fc140 msgr2=0x7f68cc104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:46:08.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:46:08.073+0000 7f68d1524700 1 -- 172.21.15.105:0/4127612518 shutdown_connections 2024-04-06T02:46:08.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:46:08.073+0000 7f68d1524700 1 -- 172.21.15.105:0/4127612518 wait complete. 2024-04-06T02:46:08.958 DEBUG:teuthology.orchestra.run.smithi105:osd.2> sudo journalctl -f -n 0 -u ceph-130df578-f3bf-11ee-b649-cb9ed24678a4@osd.2.service 2024-04-06T02:46:08.960 INFO:tasks.cephadm:Deploying osd.3 on smithi184 with /dev/vg_nvme/lv_4... 2024-04-06T02:46:08.960 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-06T02:46:09.173 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi184/config 2024-04-06T02:46:09.255 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:09 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:09.255 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:09 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:09.255 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:09 smithi105 ceph-mon[29228]: from='osd.2 [v2:172.21.15.105:6818/3651690266,v1:172.21.15.105:6819/3651690266]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-06T02:46:09.255 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:09 smithi105 ceph-mon[29228]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:46:09.295 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:09 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:09.295 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:09 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:09.295 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:09 smithi184 ceph-mon[37439]: from='osd.2 [v2:172.21.15.105:6818/3651690266,v1:172.21.15.105:6819/3651690266]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-06T02:46:09.295 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:09 smithi184 ceph-mon[37439]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:46:10.318 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:10 smithi184 ceph-mon[37439]: from='osd.2 [v2:172.21.15.105:6818/3651690266,v1:172.21.15.105:6819/3651690266]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-06T02:46:10.318 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:10 smithi184 ceph-mon[37439]: osdmap e17: 3 total, 2 up, 3 in 2024-04-06T02:46:10.318 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:10 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:46:10.319 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:10 smithi184 ceph-mon[37439]: from='osd.2 [v2:172.21.15.105:6818/3651690266,v1:172.21.15.105:6819/3651690266]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-06T02:46:10.319 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:10 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:46:10.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:10 smithi105 ceph-mon[29228]: from='osd.2 [v2:172.21.15.105:6818/3651690266,v1:172.21.15.105:6819/3651690266]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-06T02:46:10.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:10 smithi105 ceph-mon[29228]: osdmap e17: 3 total, 2 up, 3 in 2024-04-06T02:46:10.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:10 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:46:10.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:10 smithi105 ceph-mon[29228]: from='osd.2 [v2:172.21.15.105:6818/3651690266,v1:172.21.15.105:6819/3651690266]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-06T02:46:10.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:10 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:46:10.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:46:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:46:10.076+0000 7fb162c7a700 -1 osd.2 0 waiting for initial osdmap 2024-04-06T02:46:10.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:46:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:46:10.082+0000 7fb15d26c700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T02:46:11.003 INFO:teuthology.orchestra.run.smithi184.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-06T02:46:11.003 INFO:teuthology.orchestra.run.smithi184.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-06T02:46:11.003 INFO:teuthology.orchestra.run.smithi184.stderr: stderr: 10+0 records in 2024-04-06T02:46:11.003 INFO:teuthology.orchestra.run.smithi184.stderr:10+0 records out 2024-04-06T02:46:11.003 INFO:teuthology.orchestra.run.smithi184.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0171393 s, 612 MB/s 2024-04-06T02:46:11.003 INFO:teuthology.orchestra.run.smithi184.stderr:--> Zapping successful for: 2024-04-06T02:46:11.299 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:11 smithi184 ceph-mon[37439]: from='osd.2 [v2:172.21.15.105:6818/3651690266,v1:172.21.15.105:6819/3651690266]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-06T02:46:11.299 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:11 smithi184 ceph-mon[37439]: osdmap e18: 3 total, 2 up, 3 in 2024-04-06T02:46:11.299 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:11 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:46:11.299 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:11 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:46:11.299 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:11 smithi184 ceph-mon[37439]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:46:11.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:11 smithi105 ceph-mon[29228]: from='osd.2 [v2:172.21.15.105:6818/3651690266,v1:172.21.15.105:6819/3651690266]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-06T02:46:11.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:11 smithi105 ceph-mon[29228]: osdmap e18: 3 total, 2 up, 3 in 2024-04-06T02:46:11.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:11 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:46:11.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:11 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:46:11.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:11 smithi105 ceph-mon[29228]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-06T02:46:11.606 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi184:vg_nvme/lv_4 2024-04-06T02:46:11.816 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi184/config 2024-04-06T02:46:12.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:12 smithi105 ceph-mon[29228]: purged_snaps scrub starts 2024-04-06T02:46:12.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:12 smithi105 ceph-mon[29228]: purged_snaps scrub ok 2024-04-06T02:46:12.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:12 smithi105 ceph-mon[29228]: osd.2 [v2:172.21.15.105:6818/3651690266,v1:172.21.15.105:6819/3651690266] boot 2024-04-06T02:46:12.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:12 smithi105 ceph-mon[29228]: osdmap e19: 3 total, 3 up, 3 in 2024-04-06T02:46:12.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:12 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:46:12.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:12 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:12.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:12 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:12.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:12 smithi184 ceph-mon[37439]: purged_snaps scrub starts 2024-04-06T02:46:12.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:12 smithi184 ceph-mon[37439]: purged_snaps scrub ok 2024-04-06T02:46:12.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:12 smithi184 ceph-mon[37439]: osd.2 [v2:172.21.15.105:6818/3651690266,v1:172.21.15.105:6819/3651690266] boot 2024-04-06T02:46:12.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:12 smithi184 ceph-mon[37439]: osdmap e19: 3 total, 3 up, 3 in 2024-04-06T02:46:12.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:12 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T02:46:12.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:12 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:12.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:12 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:13.165 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:13 smithi184 ceph-mon[37439]: osdmap e20: 3 total, 3 up, 3 in 2024-04-06T02:46:13.166 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:13 smithi184 ceph-mon[37439]: pgmap v77: 0 pgs: ; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:46:13.166 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:13 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-06T02:46:13.168 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:13 smithi105 ceph-mon[29228]: osdmap e20: 3 total, 3 up, 3 in 2024-04-06T02:46:13.168 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:13 smithi105 ceph-mon[29228]: pgmap v77: 0 pgs: ; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:46:13.168 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:13 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-06T02:46:13.368 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.381+0000 7f3fb2c0f700 1 -- 172.21.15.184:0/3814797823 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3fac100bb0 msgr2=0x7f3fac100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:46:13.368 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.381+0000 7f3fb2c0f700 1 --2- 172.21.15.184:0/3814797823 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3fac100bb0 0x7f3fac100fd0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f3fa0009910 tx=0x7f3fa0009c20 comp rx=0 tx=0).stop 2024-04-06T02:46:13.368 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.381+0000 7f3fb2c0f700 1 -- 172.21.15.184:0/3814797823 shutdown_connections 2024-04-06T02:46:13.369 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.381+0000 7f3fb2c0f700 1 --2- 172.21.15.184:0/3814797823 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3fac101db0 0x7f3fac102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:46:13.369 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.381+0000 7f3fb2c0f700 1 --2- 172.21.15.184:0/3814797823 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3fac100bb0 0x7f3fac100fd0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:46:13.369 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.381+0000 7f3fb2c0f700 1 -- 172.21.15.184:0/3814797823 >> 172.21.15.184:0/3814797823 conn(0x7f3fac0fc110 msgr2=0x7f3fac0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:46:13.369 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.381+0000 7f3fb2c0f700 1 -- 172.21.15.184:0/3814797823 shutdown_connections 2024-04-06T02:46:13.369 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.381+0000 7f3fb2c0f700 1 -- 172.21.15.184:0/3814797823 wait complete. 2024-04-06T02:46:13.369 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.382+0000 7f3fb2c0f700 1 Processor -- start 2024-04-06T02:46:13.369 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.382+0000 7f3fb2c0f700 1 -- start start 2024-04-06T02:46:13.369 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.382+0000 7f3fb2c0f700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3fac100bb0 0x7f3fac196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:46:13.369 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.382+0000 7f3fb2c0f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3fac101db0 0x7f3fac1968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:46:13.369 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.382+0000 7f3fb2c0f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac196ee0 con 0x7f3fac101db0 2024-04-06T02:46:13.369 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.382+0000 7f3fb2c0f700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac197020 con 0x7f3fac100bb0 2024-04-06T02:46:13.369 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.382+0000 7f3fb09ab700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3fac100bb0 0x7f3fac196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:46:13.369 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.382+0000 7f3fb09ab700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3fac100bb0 0x7f3fac196380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:45452/0 (socket says 172.21.15.184:45452) 2024-04-06T02:46:13.370 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.382+0000 7f3fb09ab700 1 -- 172.21.15.184:0/534860598 learned_addr learned my addr 172.21.15.184:0/534860598 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-04-06T02:46:13.370 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.382+0000 7f3fabfff700 1 --2- 172.21.15.184:0/534860598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3fac101db0 0x7f3fac1968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:46:13.370 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.382+0000 7f3fb09ab700 1 -- 172.21.15.184:0/534860598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3fac101db0 msgr2=0x7f3fac1968c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:46:13.370 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.382+0000 7f3fb09ab700 1 --2- 172.21.15.184:0/534860598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3fac101db0 0x7f3fac1968c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:46:13.370 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.382+0000 7f3fb09ab700 1 -- 172.21.15.184:0/534860598 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa00095f0 con 0x7f3fac100bb0 2024-04-06T02:46:13.370 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.382+0000 7f3fabfff700 1 --2- 172.21.15.184:0/534860598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3fac101db0 0x7f3fac1968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T02:46:13.370 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.382+0000 7f3fb09ab700 1 --2- 172.21.15.184:0/534860598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3fac100bb0 0x7f3fac196380 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3fa000f690 tx=0x7f3fa000f770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:46:13.370 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.382+0000 7f3fa9ffb700 1 -- 172.21.15.184:0/534860598 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fa001d070 con 0x7f3fac100bb0 2024-04-06T02:46:13.370 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.383+0000 7f3fa9ffb700 1 -- 172.21.15.184:0/534860598 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3fa0005dc0 con 0x7f3fac100bb0 2024-04-06T02:46:13.370 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.383+0000 7f3fb2c0f700 1 -- 172.21.15.184:0/534860598 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fac19ba70 con 0x7f3fac100bb0 2024-04-06T02:46:13.370 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.383+0000 7f3fa9ffb700 1 -- 172.21.15.184:0/534860598 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fa0017630 con 0x7f3fac100bb0 2024-04-06T02:46:13.370 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.383+0000 7f3fb2c0f700 1 -- 172.21.15.184:0/534860598 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fac19bf60 con 0x7f3fac100bb0 2024-04-06T02:46:13.371 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.384+0000 7f3fb2c0f700 1 -- 172.21.15.184:0/534860598 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fac109710 con 0x7f3fac100bb0 2024-04-06T02:46:13.371 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.384+0000 7f3fa9ffb700 1 -- 172.21.15.184:0/534860598 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7f3fa0003680 con 0x7f3fac100bb0 2024-04-06T02:46:13.372 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.385+0000 7f3fa9ffb700 1 --2- 172.21.15.184:0/534860598 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f3f9c06c470 0x7f3f9c06e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:46:13.372 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.385+0000 7f3fa9ffb700 1 -- 172.21.15.184:0/534860598 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3221+0+0 (secure 0 0 0) 0x7f3fa008c350 con 0x7f3fac100bb0 2024-04-06T02:46:13.372 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.385+0000 7f3fabfff700 1 --2- 172.21.15.184:0/534860598 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f3f9c06c470 0x7f3f9c06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:46:13.372 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.385+0000 7f3fabfff700 1 --2- 172.21.15.184:0/534860598 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f3f9c06c470 0x7f3f9c06e930 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3f98009cb0 tx=0x7f3f98009260 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:46:13.374 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.389+0000 7f3fa9ffb700 1 -- 172.21.15.184:0/534860598 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3fa005af60 con 0x7f3fac100bb0 2024-04-06T02:46:13.575 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:13.590+0000 7f3fb2c0f700 1 -- 172.21.15.184:0/534860598 --> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f3fac19c660 con 0x7f3f9c06c470 2024-04-06T02:46:14.620 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:14 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-06T02:46:14.621 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:14 smithi105 ceph-mon[29228]: osdmap e21: 3 total, 3 up, 3 in 2024-04-06T02:46:14.621 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:14 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T02:46:14.621 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:14 smithi105 ceph-mon[29228]: from='client.24123 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:46:14.621 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:14 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:46:14.621 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:14 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:46:14.621 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:14 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:46:14.711 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:14 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-06T02:46:14.711 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:14 smithi184 ceph-mon[37439]: osdmap e21: 3 total, 3 up, 3 in 2024-04-06T02:46:14.711 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:14 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T02:46:14.711 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:14 smithi184 ceph-mon[37439]: from='client.24123 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:46:14.711 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:14 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:46:14.711 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:14 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:46:14.711 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:14 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:46:15.674 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:46:15 smithi105 sudo[67359]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-06T02:46:15.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:46:15 smithi105 sudo[67359]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-06T02:46:15.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:46:15 smithi105 sudo[67359]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-06T02:46:15.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:46:15 smithi105 sudo[67359]: pam_unix(sudo:session): session closed for user root 2024-04-06T02:46:15.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:15 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-06T02:46:15.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:15 smithi105 ceph-mon[29228]: osdmap e22: 3 total, 3 up, 3 in 2024-04-06T02:46:15.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:15 smithi105 ceph-mon[29228]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:46:15.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:15 smithi105 ceph-mon[29228]: Detected new or changed devices on smithi105 2024-04-06T02:46:15.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:15 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:15.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:15 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:15.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:15 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:46:15.676 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:15 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:46:15.676 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:15 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:46:15.676 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:15 smithi105 ceph-mon[29228]: Adjusting osd_memory_target on smithi105 to 3946M 2024-04-06T02:46:15.676 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:15 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:15.676 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:15 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:46:15.676 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:15 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:46:15.676 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:15 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:15.676 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:15 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:46:15.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:15 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-06T02:46:15.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:15 smithi184 ceph-mon[37439]: osdmap e22: 3 total, 3 up, 3 in 2024-04-06T02:46:15.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:15 smithi184 ceph-mon[37439]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:46:15.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:15 smithi184 ceph-mon[37439]: Detected new or changed devices on smithi105 2024-04-06T02:46:15.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:15 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:15.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:15 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:15.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:15 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:46:15.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:15 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:46:15.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:15 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:46:15.732 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:15 smithi184 ceph-mon[37439]: Adjusting osd_memory_target on smithi105 to 3946M 2024-04-06T02:46:15.732 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:15 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:15.732 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:15 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:46:15.732 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:15 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:46:15.732 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:15 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:15.732 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:15 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:46:16.134 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:46:15 smithi105 sudo[67380]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-06T02:46:16.134 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:46:15 smithi105 sudo[67380]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-06T02:46:16.134 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:46:15 smithi105 sudo[67380]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-06T02:46:16.134 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:46:15 smithi105 sudo[67380]: pam_unix(sudo:session): session closed for user root 2024-04-06T02:46:16.363 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:16.378+0000 7f3fa9ffb700 1 -- 172.21.15.184:0/534860598 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f3fa00578a0 con 0x7f3fac100bb0 2024-04-06T02:46:16.402 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:46:16 smithi105 sudo[67463]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-06T02:46:16.402 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:46:16 smithi105 sudo[67463]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-06T02:46:16.402 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:46:16 smithi105 sudo[67463]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-06T02:46:16.402 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:46:16 smithi105 sudo[67463]: pam_unix(sudo:session): session closed for user root 2024-04-06T02:46:16.402 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:46:16 smithi105 sudo[67488]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-06T02:46:16.665 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:46:16 smithi105 sudo[67488]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-06T02:46:16.665 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:46:16 smithi105 sudo[67488]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-06T02:46:16.665 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:46:16 smithi105 sudo[67488]: pam_unix(sudo:session): session closed for user root 2024-04-06T02:46:16.665 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:46:16 smithi105 sudo[67491]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-06T02:46:16.665 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:46:16 smithi105 sudo[67491]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-06T02:46:16.665 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:46:16 smithi105 sudo[67491]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-06T02:46:16.665 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:46:16 smithi105 sudo[67491]: pam_unix(sudo:session): session closed for user root 2024-04-06T02:46:16.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:16 smithi105 ceph-mon[29228]: osdmap e23: 3 total, 3 up, 3 in 2024-04-06T02:46:16.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:16 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:16.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:16 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/1709653212' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97f959d7-5922-4423-9d63-e2fe677a7019"}]: dispatch 2024-04-06T02:46:16.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:16 smithi105 ceph-mon[29228]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97f959d7-5922-4423-9d63-e2fe677a7019"}]: dispatch 2024-04-06T02:46:16.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:16 smithi105 ceph-mon[29228]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97f959d7-5922-4423-9d63-e2fe677a7019"}]': finished 2024-04-06T02:46:16.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:16 smithi105 ceph-mon[29228]: osdmap e24: 4 total, 3 up, 4 in 2024-04-06T02:46:16.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:16 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:46:16.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:16 smithi105 ceph-mon[29228]: mgrmap e19: smithi105.vqqcpx(active, since 2m), standbys: smithi184.fzhhql 2024-04-06T02:46:16.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:16 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/2349599600' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:46:16.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:16 smithi184 ceph-mon[37439]: osdmap e23: 3 total, 3 up, 3 in 2024-04-06T02:46:16.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:16 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:16.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:16 smithi184 ceph-mon[37439]: from='client.? 172.21.15.184:0/1709653212' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97f959d7-5922-4423-9d63-e2fe677a7019"}]: dispatch 2024-04-06T02:46:16.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:16 smithi184 ceph-mon[37439]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97f959d7-5922-4423-9d63-e2fe677a7019"}]: dispatch 2024-04-06T02:46:16.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:16 smithi184 ceph-mon[37439]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97f959d7-5922-4423-9d63-e2fe677a7019"}]': finished 2024-04-06T02:46:16.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:16 smithi184 ceph-mon[37439]: osdmap e24: 4 total, 3 up, 4 in 2024-04-06T02:46:16.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:16 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:46:16.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:16 smithi184 ceph-mon[37439]: mgrmap e19: smithi105.vqqcpx(active, since 2m), standbys: smithi184.fzhhql 2024-04-06T02:46:16.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:16 smithi184 ceph-mon[37439]: from='client.? 172.21.15.184:0/2349599600' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:46:17.177 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:17 smithi105 sudo[67570]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-06T02:46:17.178 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:46:16 smithi105 sudo[67541]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-06T02:46:17.178 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:46:16 smithi105 sudo[67541]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-06T02:46:17.178 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:46:16 smithi105 sudo[67541]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-06T02:46:17.178 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:46:16 smithi105 sudo[67541]: pam_unix(sudo:session): session closed for user root 2024-04-06T02:46:17.434 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:17 smithi105 sudo[67570]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-06T02:46:17.434 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:17 smithi105 sudo[67570]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-06T02:46:17.434 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:17 smithi105 sudo[67570]: pam_unix(sudo:session): session closed for user root 2024-04-06T02:46:17.783 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:17 smithi105 sudo[67573]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-06T02:46:17.783 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:17 smithi105 sudo[67573]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-06T02:46:17.783 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:17 smithi105 sudo[67573]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-06T02:46:17.783 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:17 smithi105 sudo[67573]: pam_unix(sudo:session): session closed for user root 2024-04-06T02:46:17.941 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:17 smithi184 sudo[41658]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-06T02:46:17.941 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:17 smithi184 sudo[41658]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-06T02:46:17.941 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:17 smithi184 sudo[41658]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-06T02:46:17.942 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:17 smithi184 sudo[41658]: pam_unix(sudo:session): session closed for user root 2024-04-06T02:46:18.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:17 smithi105 ceph-mon[29228]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:46:18.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:17 smithi105 ceph-mon[29228]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T02:46:18.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:17 smithi105 ceph-mon[29228]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T02:46:18.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:17 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-04-06T02:46:18.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:17 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-04-06T02:46:18.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:17 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-04-06T02:46:18.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:17 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-04-06T02:46:18.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:17 smithi184 sudo[41845]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-06T02:46:18.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:17 smithi184 sudo[41845]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-06T02:46:18.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:17 smithi184 sudo[41845]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-06T02:46:18.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:17 smithi184 sudo[41845]: pam_unix(sudo:session): session closed for user root 2024-04-06T02:46:18.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:17 smithi184 ceph-mon[37439]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:46:18.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:17 smithi184 ceph-mon[37439]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T02:46:18.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:17 smithi184 ceph-mon[37439]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T02:46:18.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:17 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-04-06T02:46:18.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:17 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-04-06T02:46:18.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:17 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-04-06T02:46:18.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:17 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mon metadata", "id": "smithi184"}]: dispatch 2024-04-06T02:46:19.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:18 smithi184 ceph-mon[37439]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T02:46:19.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:18 smithi184 ceph-mon[37439]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T02:46:19.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:18 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:19.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:18 smithi184 ceph-mon[37439]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:46:19.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:18 smithi105 ceph-mon[29228]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T02:46:19.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:18 smithi105 ceph-mon[29228]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T02:46:19.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:18 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:19.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:18 smithi105 ceph-mon[29228]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:46:20.221 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:20 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:20.221 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:20 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:20.221 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:20 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:46:20.221 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:20 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:46:20.221 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:20 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:20.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:20 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:20.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:20 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:20.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:20 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:46:20.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:20 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:46:20.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:20 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:21.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:21 smithi184 ceph-mon[37439]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:46:21.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:21 smithi105 ceph-mon[29228]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:46:23.702 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:23 smithi184 ceph-mon[37439]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-04-06T02:46:23.702 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:23 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-06T02:46:23.702 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:23 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:46:23.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:23 smithi105 ceph-mon[29228]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-04-06T02:46:23.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:23 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-06T02:46:23.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:23 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:46:24.711 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:24 smithi105 ceph-mon[29228]: Deploying daemon osd.3 on smithi184 2024-04-06T02:46:24.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:24 smithi184 ceph-mon[37439]: Deploying daemon osd.3 on smithi184 2024-04-06T02:46:25.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:25 smithi184 ceph-mon[37439]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-06T02:46:25.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:25 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:46:25.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:25 smithi105 ceph-mon[29228]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-06T02:46:25.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:25 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:46:27.905 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:27 smithi184 ceph-mon[37439]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-06T02:46:27.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:27 smithi105 ceph-mon[29228]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-06T02:46:29.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:29 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:29.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:29 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:29.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:29 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:46:29.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:29 smithi184 ceph-mon[37439]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-06T02:46:29.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:29 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:29.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:29 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:29.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:29 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:46:29.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:29 smithi105 ceph-mon[29228]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-06T02:46:31.330 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:31.343+0000 7f3fa9ffb700 1 -- 172.21.15.184:0/534860598 <== mgr.14221 v2:172.21.15.105:6800/2334550011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3fac19c660 con 0x7f3f9c06c470 2024-04-06T02:46:31.330 INFO:teuthology.orchestra.run.smithi184.stdout:Created osd(s) 3 on host 'smithi184' 2024-04-06T02:46:31.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:31.347+0000 7f3fb2c0f700 1 -- 172.21.15.184:0/534860598 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f3f9c06c470 msgr2=0x7f3f9c06e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:46:31.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:31.347+0000 7f3fb2c0f700 1 --2- 172.21.15.184:0/534860598 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f3f9c06c470 0x7f3f9c06e930 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3f98009cb0 tx=0x7f3f98009260 comp rx=0 tx=0).stop 2024-04-06T02:46:31.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:31.347+0000 7f3fb2c0f700 1 -- 172.21.15.184:0/534860598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3fac100bb0 msgr2=0x7f3fac196380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:46:31.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:31.347+0000 7f3fb2c0f700 1 --2- 172.21.15.184:0/534860598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3fac100bb0 0x7f3fac196380 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3fa000f690 tx=0x7f3fa000f770 comp rx=0 tx=0).stop 2024-04-06T02:46:31.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:31.347+0000 7f3fb2c0f700 1 -- 172.21.15.184:0/534860598 shutdown_connections 2024-04-06T02:46:31.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:31.347+0000 7f3fb2c0f700 1 --2- 172.21.15.184:0/534860598 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f3f9c06c470 0x7f3f9c06e930 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:46:31.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:31.347+0000 7f3fb2c0f700 1 --2- 172.21.15.184:0/534860598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3fac100bb0 0x7f3fac196380 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:46:31.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:31.347+0000 7f3fb2c0f700 1 --2- 172.21.15.184:0/534860598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3fac101db0 0x7f3fac1968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:46:31.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:31.347+0000 7f3fb2c0f700 1 -- 172.21.15.184:0/534860598 >> 172.21.15.184:0/534860598 conn(0x7f3fac0fc110 msgr2=0x7f3fac104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:46:31.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:31.347+0000 7f3fb2c0f700 1 -- 172.21.15.184:0/534860598 shutdown_connections 2024-04-06T02:46:31.333 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:31.347+0000 7f3fb2c0f700 1 -- 172.21.15.184:0/534860598 wait complete. 2024-04-06T02:46:31.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:31 smithi105 ceph-mon[29228]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-06T02:46:31.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:31 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:31.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:31 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:31.963 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:31 smithi184 ceph-mon[37439]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-06T02:46:31.963 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:31 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:31.963 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:31 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:32.546 DEBUG:teuthology.orchestra.run.smithi184:osd.3> sudo journalctl -f -n 0 -u ceph-130df578-f3bf-11ee-b649-cb9ed24678a4@osd.3.service 2024-04-06T02:46:32.548 INFO:tasks.cephadm:Deploying osd.4 on smithi184 with /dev/vg_nvme/lv_3... 2024-04-06T02:46:32.548 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-06T02:46:32.808 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi184/config 2024-04-06T02:46:32.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:32 smithi105 ceph-mon[29228]: from='osd.3 [v2:172.21.15.184:6800/330017535,v1:172.21.15.184:6801/330017535]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T02:46:32.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:32 smithi105 ceph-mon[29228]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T02:46:32.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:32 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:32.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:32 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:32.930 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:32 smithi184 ceph-mon[37439]: from='osd.3 [v2:172.21.15.184:6800/330017535,v1:172.21.15.184:6801/330017535]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T02:46:32.930 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:32 smithi184 ceph-mon[37439]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T02:46:32.930 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:32 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:32.930 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:32 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:33.906 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:33 smithi184 ceph-mon[37439]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-06T02:46:33.907 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:33 smithi184 ceph-mon[37439]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-06T02:46:33.907 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:33 smithi184 ceph-mon[37439]: osdmap e25: 4 total, 3 up, 4 in 2024-04-06T02:46:33.907 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:33 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:46:33.907 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:33 smithi184 ceph-mon[37439]: from='osd.3 [v2:172.21.15.184:6800/330017535,v1:172.21.15.184:6801/330017535]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-04-06T02:46:33.907 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:33 smithi184 ceph-mon[37439]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-04-06T02:46:33.907 INFO:journalctl@ceph.osd.3.smithi184.stdout:Apr 06 02:46:33 smithi184 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-3[43471]: 2024-04-06T02:46:33.672+0000 7f5399dfe700 -1 osd.3 0 waiting for initial osdmap 2024-04-06T02:46:33.907 INFO:journalctl@ceph.osd.3.smithi184.stdout:Apr 06 02:46:33 smithi184 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-3[43471]: 2024-04-06T02:46:33.677+0000 7f5392bed700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T02:46:33.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:33 smithi105 ceph-mon[29228]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-06T02:46:33.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:33 smithi105 ceph-mon[29228]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-06T02:46:33.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:33 smithi105 ceph-mon[29228]: osdmap e25: 4 total, 3 up, 4 in 2024-04-06T02:46:33.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:33 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:46:33.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:33 smithi105 ceph-mon[29228]: from='osd.3 [v2:172.21.15.184:6800/330017535,v1:172.21.15.184:6801/330017535]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-04-06T02:46:33.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:33 smithi105 ceph-mon[29228]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-04-06T02:46:34.669 INFO:teuthology.orchestra.run.smithi184.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-06T02:46:34.669 INFO:teuthology.orchestra.run.smithi184.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-06T02:46:34.670 INFO:teuthology.orchestra.run.smithi184.stderr: stderr: 10+0 records in 2024-04-06T02:46:34.670 INFO:teuthology.orchestra.run.smithi184.stderr:10+0 records out 2024-04-06T02:46:34.670 INFO:teuthology.orchestra.run.smithi184.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0145865 s, 719 MB/s 2024-04-06T02:46:34.670 INFO:teuthology.orchestra.run.smithi184.stderr:--> Zapping successful for: 2024-04-06T02:46:34.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:34 smithi184 ceph-mon[37439]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-04-06T02:46:34.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:34 smithi184 ceph-mon[37439]: osdmap e26: 4 total, 3 up, 4 in 2024-04-06T02:46:34.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:34 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:46:34.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:34 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:46:34.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:34 smithi105 ceph-mon[29228]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-04-06T02:46:34.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:34 smithi105 ceph-mon[29228]: osdmap e26: 4 total, 3 up, 4 in 2024-04-06T02:46:34.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:34 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:46:34.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:34 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:46:35.257 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi184:vg_nvme/lv_3 2024-04-06T02:46:35.906 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:35 smithi184 ceph-mon[37439]: purged_snaps scrub starts 2024-04-06T02:46:35.906 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:35 smithi184 ceph-mon[37439]: purged_snaps scrub ok 2024-04-06T02:46:35.906 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:35 smithi184 ceph-mon[37439]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:46:35.906 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:35 smithi184 ceph-mon[37439]: osd.3 [v2:172.21.15.184:6800/330017535,v1:172.21.15.184:6801/330017535] boot 2024-04-06T02:46:35.906 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:35 smithi184 ceph-mon[37439]: osdmap e27: 4 total, 4 up, 4 in 2024-04-06T02:46:35.906 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:35 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:46:35.906 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:35 smithi184 ceph-mon[37439]: osdmap e28: 4 total, 4 up, 4 in 2024-04-06T02:46:35.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:35 smithi105 ceph-mon[29228]: purged_snaps scrub starts 2024-04-06T02:46:35.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:35 smithi105 ceph-mon[29228]: purged_snaps scrub ok 2024-04-06T02:46:35.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:35 smithi105 ceph-mon[29228]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-06T02:46:35.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:35 smithi105 ceph-mon[29228]: osd.3 [v2:172.21.15.184:6800/330017535,v1:172.21.15.184:6801/330017535] boot 2024-04-06T02:46:35.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:35 smithi105 ceph-mon[29228]: osdmap e27: 4 total, 4 up, 4 in 2024-04-06T02:46:35.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:35 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T02:46:35.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:35 smithi105 ceph-mon[29228]: osdmap e28: 4 total, 4 up, 4 in 2024-04-06T02:46:35.996 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi184/config 2024-04-06T02:46:37.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:37 smithi184 ceph-mon[37439]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:46:37.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:37 smithi184 ceph-mon[37439]: osdmap e29: 4 total, 4 up, 4 in 2024-04-06T02:46:37.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:37 smithi184 ceph-mon[37439]: osdmap e30: 4 total, 4 up, 4 in 2024-04-06T02:46:37.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:37 smithi105 ceph-mon[29228]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:46:37.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:37 smithi105 ceph-mon[29228]: osdmap e29: 4 total, 4 up, 4 in 2024-04-06T02:46:37.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:37 smithi105 ceph-mon[29228]: osdmap e30: 4 total, 4 up, 4 in 2024-04-06T02:46:38.494 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.507+0000 7f3bcea8f700 1 -- 172.21.15.184:0/3339124263 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3bc8101dc0 msgr2=0x7f3bc8102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:46:38.495 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.507+0000 7f3bcea8f700 1 --2- 172.21.15.184:0/3339124263 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3bc8101dc0 0x7f3bc8102220 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4009910 tx=0x7f3bb4009c20 comp rx=0 tx=0).stop 2024-04-06T02:46:38.495 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.507+0000 7f3bcea8f700 1 -- 172.21.15.184:0/3339124263 shutdown_connections 2024-04-06T02:46:38.495 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.507+0000 7f3bcea8f700 1 --2- 172.21.15.184:0/3339124263 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3bc8101dc0 0x7f3bc8102220 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:46:38.495 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.507+0000 7f3bcea8f700 1 --2- 172.21.15.184:0/3339124263 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3bc8100bc0 0x7f3bc8100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:46:38.495 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.507+0000 7f3bcea8f700 1 -- 172.21.15.184:0/3339124263 >> 172.21.15.184:0/3339124263 conn(0x7f3bc80fc140 msgr2=0x7f3bc80fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:46:38.495 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.507+0000 7f3bcea8f700 1 -- 172.21.15.184:0/3339124263 shutdown_connections 2024-04-06T02:46:38.495 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.507+0000 7f3bcea8f700 1 -- 172.21.15.184:0/3339124263 wait complete. 2024-04-06T02:46:38.495 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.508+0000 7f3bcea8f700 1 Processor -- start 2024-04-06T02:46:38.495 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.508+0000 7f3bcea8f700 1 -- start start 2024-04-06T02:46:38.495 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.508+0000 7f3bcea8f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3bc8100bc0 0x7f3bc8074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:46:38.495 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.508+0000 7f3bcea8f700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3bc8101dc0 0x7f3bc8073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:46:38.496 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.508+0000 7f3bcea8f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc80736c0 con 0x7f3bc8100bc0 2024-04-06T02:46:38.496 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.508+0000 7f3bcea8f700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc8073800 con 0x7f3bc8101dc0 2024-04-06T02:46:38.496 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.508+0000 7f3bc7fff700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3bc8101dc0 0x7f3bc8073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:46:38.496 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.508+0000 7f3bc7fff700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3bc8101dc0 0x7f3bc8073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:51300/0 (socket says 172.21.15.184:51300) 2024-04-06T02:46:38.496 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.508+0000 7f3bc7fff700 1 -- 172.21.15.184:0/1226098455 learned_addr learned my addr 172.21.15.184:0/1226098455 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-04-06T02:46:38.496 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.508+0000 7f3bcc82b700 1 --2- 172.21.15.184:0/1226098455 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3bc8100bc0 0x7f3bc8074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:46:38.496 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.508+0000 7f3bc7fff700 1 -- 172.21.15.184:0/1226098455 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3bc8100bc0 msgr2=0x7f3bc8074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:46:38.496 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.508+0000 7f3bc7fff700 1 --2- 172.21.15.184:0/1226098455 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3bc8100bc0 0x7f3bc8074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:46:38.496 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.508+0000 7f3bc7fff700 1 -- 172.21.15.184:0/1226098455 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb40095f0 con 0x7f3bc8101dc0 2024-04-06T02:46:38.496 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.508+0000 7f3bc7fff700 1 --2- 172.21.15.184:0/1226098455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3bc8101dc0 0x7f3bc8073180 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3bb400bd70 tx=0x7f3bb400be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:46:38.496 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.508+0000 7f3bc5ffb700 1 -- 172.21.15.184:0/1226098455 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3bb401d070 con 0x7f3bc8101dc0 2024-04-06T02:46:38.496 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.508+0000 7f3bcea8f700 1 -- 172.21.15.184:0/1226098455 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bc8073a80 con 0x7f3bc8101dc0 2024-04-06T02:46:38.497 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.508+0000 7f3bc5ffb700 1 -- 172.21.15.184:0/1226098455 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3bb4022470 con 0x7f3bc8101dc0 2024-04-06T02:46:38.497 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.509+0000 7f3bc5ffb700 1 -- 172.21.15.184:0/1226098455 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3bb400f670 con 0x7f3bc8101dc0 2024-04-06T02:46:38.497 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.509+0000 7f3bcea8f700 1 -- 172.21.15.184:0/1226098455 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bc8073f70 con 0x7f3bc8101dc0 2024-04-06T02:46:38.498 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.510+0000 7f3bcea8f700 1 -- 172.21.15.184:0/1226098455 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bc8109720 con 0x7f3bc8101dc0 2024-04-06T02:46:38.498 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.510+0000 7f3bc5ffb700 1 -- 172.21.15.184:0/1226098455 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f3bb4022a50 con 0x7f3bc8101dc0 2024-04-06T02:46:38.498 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.510+0000 7f3bc5ffb700 1 --2- 172.21.15.184:0/1226098455 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f3bb806c4e0 0x7f3bb806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:46:38.498 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.511+0000 7f3bc5ffb700 1 -- 172.21.15.184:0/1226098455 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 3774+0+0 (secure 0 0 0) 0x7f3bb408c840 con 0x7f3bc8101dc0 2024-04-06T02:46:38.498 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.511+0000 7f3bcc82b700 1 --2- 172.21.15.184:0/1226098455 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f3bb806c4e0 0x7f3bb806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:46:38.498 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.511+0000 7f3bcc82b700 1 --2- 172.21.15.184:0/1226098455 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f3bb806c4e0 0x7f3bb806e9a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc005fd0 tx=0x7f3bbc005e50 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:46:38.500 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.515+0000 7f3bc5ffb700 1 -- 172.21.15.184:0/1226098455 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3bb405b200 con 0x7f3bc8101dc0 2024-04-06T02:46:38.698 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:38.712+0000 7f3bcea8f700 1 -- 172.21.15.184:0/1226098455 --> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f3bc8106710 con 0x7f3bb806c4e0 2024-04-06T02:46:39.342 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:39 smithi184 ceph-mon[37439]: Detected new or changed devices on smithi184 2024-04-06T02:46:39.343 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:39 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:39.343 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:39 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:39.343 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:39 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:46:39.343 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:39 smithi184 ceph-mon[37439]: Adjusting osd_memory_target on smithi184 to 14912M 2024-04-06T02:46:39.343 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:39 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:39.343 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:39 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:46:39.343 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:39 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:46:39.343 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:39 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:39.343 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:39 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:46:39.343 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:39 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:46:39.343 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:39 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:46:39.343 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:39 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:39.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:39 smithi184 ceph-mon[37439]: osdmap e31: 4 total, 4 up, 4 in 2024-04-06T02:46:39.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:39 smithi184 ceph-mon[37439]: pgmap v101: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 146 KiB/s, 0 objects/s recovering 2024-04-06T02:46:39.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:39 smithi184 ceph-mon[37439]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:46:39.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:39 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:46:39.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:39 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:46:39.344 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:39 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:46:39.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:39 smithi105 ceph-mon[29228]: Detected new or changed devices on smithi184 2024-04-06T02:46:39.467 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:39 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:39.467 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:39 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:39.467 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:39 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:46:39.467 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:39 smithi105 ceph-mon[29228]: Adjusting osd_memory_target on smithi184 to 14912M 2024-04-06T02:46:39.467 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:39 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:39.468 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:39 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:46:39.468 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:39 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:46:39.468 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:39 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:39.468 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:39 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:46:39.468 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:39 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:46:39.468 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:39 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:46:39.468 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:39 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:39.468 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:39 smithi105 ceph-mon[29228]: osdmap e31: 4 total, 4 up, 4 in 2024-04-06T02:46:39.468 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:39 smithi105 ceph-mon[29228]: pgmap v101: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 146 KiB/s, 0 objects/s recovering 2024-04-06T02:46:39.468 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:39 smithi105 ceph-mon[29228]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:46:39.468 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:39 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:46:39.468 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:39 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:46:39.468 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:39 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:46:40.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:40 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:46:40.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:40 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:46:41.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:41 smithi184 ceph-mon[37439]: pgmap v102: 1 pgs: 1 peering; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-04-06T02:46:41.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:41 smithi184 ceph-mon[37439]: from='client.? 172.21.15.184:0/1495599653' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34d7ebbd-c725-4948-ae33-3ffb40ec9979"}]: dispatch 2024-04-06T02:46:41.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:41 smithi184 ceph-mon[37439]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34d7ebbd-c725-4948-ae33-3ffb40ec9979"}]: dispatch 2024-04-06T02:46:41.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:41 smithi184 ceph-mon[37439]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34d7ebbd-c725-4948-ae33-3ffb40ec9979"}]': finished 2024-04-06T02:46:41.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:41 smithi184 ceph-mon[37439]: osdmap e32: 5 total, 4 up, 5 in 2024-04-06T02:46:41.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:41 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:46:41.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:41 smithi105 ceph-mon[29228]: pgmap v102: 1 pgs: 1 peering; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-04-06T02:46:41.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:41 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/1495599653' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34d7ebbd-c725-4948-ae33-3ffb40ec9979"}]: dispatch 2024-04-06T02:46:41.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:41 smithi105 ceph-mon[29228]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34d7ebbd-c725-4948-ae33-3ffb40ec9979"}]: dispatch 2024-04-06T02:46:41.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:41 smithi105 ceph-mon[29228]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34d7ebbd-c725-4948-ae33-3ffb40ec9979"}]': finished 2024-04-06T02:46:41.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:41 smithi105 ceph-mon[29228]: osdmap e32: 5 total, 4 up, 5 in 2024-04-06T02:46:41.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:41 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:46:42.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:42 smithi184 ceph-mon[37439]: from='client.? 172.21.15.184:0/787989866' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:46:42.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:42 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/787989866' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:46:43.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:43 smithi184 ceph-mon[37439]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-06T02:46:43.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:43 smithi105 ceph-mon[29228]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-06T02:46:45.729 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:45 smithi184 ceph-mon[37439]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-06T02:46:45.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:45 smithi105 ceph-mon[29228]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-06T02:46:47.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:47 smithi184 ceph-mon[37439]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:46:47.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:47 smithi105 ceph-mon[29228]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:46:48.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:48 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-06T02:46:48.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:48 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:46:48.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:48 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-06T02:46:48.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:48 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:46:49.905 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:49 smithi105 ceph-mon[29228]: Deploying daemon osd.4 on smithi184 2024-04-06T02:46:49.905 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:49 smithi105 ceph-mon[29228]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:46:49.945 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:49 smithi184 ceph-mon[37439]: Deploying daemon osd.4 on smithi184 2024-04-06T02:46:49.945 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:49 smithi184 ceph-mon[37439]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:46:51.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:51 smithi105 ceph-mon[29228]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:46:51.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:51 smithi184 ceph-mon[37439]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:46:53.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:53 smithi184 ceph-mon[37439]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:46:53.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:53 smithi105 ceph-mon[29228]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:46:54.866 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:54 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:54.866 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:54 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:54.867 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:54 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:46:54.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:54 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:54.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:54 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:54.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:54 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:46:55.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:55 smithi105 ceph-mon[29228]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:46:55.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:55 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:46:55.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:55 smithi184 ceph-mon[37439]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:46:55.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:55 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:46:56.356 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:56.370+0000 7f3bc5ffb700 1 -- 172.21.15.184:0/1226098455 <== mgr.14221 v2:172.21.15.105:6800/2334550011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3bc8106710 con 0x7f3bb806c4e0 2024-04-06T02:46:56.356 INFO:teuthology.orchestra.run.smithi184.stdout:Created osd(s) 4 on host 'smithi184' 2024-04-06T02:46:56.358 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:56.373+0000 7f3bcea8f700 1 -- 172.21.15.184:0/1226098455 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f3bb806c4e0 msgr2=0x7f3bb806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:46:56.358 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:56.373+0000 7f3bcea8f700 1 --2- 172.21.15.184:0/1226098455 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f3bb806c4e0 0x7f3bb806e9a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc005fd0 tx=0x7f3bbc005e50 comp rx=0 tx=0).stop 2024-04-06T02:46:56.359 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:56.373+0000 7f3bcea8f700 1 -- 172.21.15.184:0/1226098455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3bc8101dc0 msgr2=0x7f3bc8073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:46:56.359 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:56.373+0000 7f3bcea8f700 1 --2- 172.21.15.184:0/1226098455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3bc8101dc0 0x7f3bc8073180 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3bb400bd70 tx=0x7f3bb400be50 comp rx=0 tx=0).stop 2024-04-06T02:46:56.359 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:56.373+0000 7f3bcea8f700 1 -- 172.21.15.184:0/1226098455 shutdown_connections 2024-04-06T02:46:56.359 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:56.373+0000 7f3bcea8f700 1 --2- 172.21.15.184:0/1226098455 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f3bb806c4e0 0x7f3bb806e9a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:46:56.359 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:56.373+0000 7f3bcea8f700 1 --2- 172.21.15.184:0/1226098455 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3bc8100bc0 0x7f3bc8074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:46:56.359 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:56.373+0000 7f3bcea8f700 1 --2- 172.21.15.184:0/1226098455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3bc8101dc0 0x7f3bc8073180 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:46:56.359 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:56.373+0000 7f3bcea8f700 1 -- 172.21.15.184:0/1226098455 >> 172.21.15.184:0/1226098455 conn(0x7f3bc80fc140 msgr2=0x7f3bc8104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:46:56.359 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:56.373+0000 7f3bcea8f700 1 -- 172.21.15.184:0/1226098455 shutdown_connections 2024-04-06T02:46:56.359 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:46:56.373+0000 7f3bcea8f700 1 -- 172.21.15.184:0/1226098455 wait complete. 2024-04-06T02:46:57.235 DEBUG:teuthology.orchestra.run.smithi184:osd.4> sudo journalctl -f -n 0 -u ceph-130df578-f3bf-11ee-b649-cb9ed24678a4@osd.4.service 2024-04-06T02:46:57.237 INFO:tasks.cephadm:Deploying osd.5 on smithi184 with /dev/vg_nvme/lv_2... 2024-04-06T02:46:57.238 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-06T02:46:57.467 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:57 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:57.467 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:57 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:57.467 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:57 smithi184 ceph-mon[37439]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:46:57.468 INFO:journalctl@ceph.osd.4.smithi184.stdout:Apr 06 02:46:57 smithi184 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-4[50341]: 2024-04-06T02:46:57.454+0000 7f7067e1b640 -1 osd.4 0 log_to_monitors true 2024-04-06T02:46:57.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:57 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:57.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:57 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:57.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:57 smithi105 ceph-mon[29228]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:46:58.048 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi184/config 2024-04-06T02:46:58.478 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:58 smithi184 ceph-mon[37439]: from='osd.4 [v2:172.21.15.184:6808/2237051966,v1:172.21.15.184:6809/2237051966]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T02:46:58.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:58 smithi105 ceph-mon[29228]: from='osd.4 [v2:172.21.15.184:6808/2237051966,v1:172.21.15.184:6809/2237051966]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T02:46:59.551 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:59 smithi184 ceph-mon[37439]: from='osd.4 [v2:172.21.15.184:6808/2237051966,v1:172.21.15.184:6809/2237051966]' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-06T02:46:59.551 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:59 smithi184 ceph-mon[37439]: osdmap e33: 5 total, 4 up, 5 in 2024-04-06T02:46:59.551 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:59 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:46:59.551 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:59 smithi184 ceph-mon[37439]: from='osd.4 [v2:172.21.15.184:6808/2237051966,v1:172.21.15.184:6809/2237051966]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-04-06T02:46:59.551 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:59 smithi184 ceph-mon[37439]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:46:59.551 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:59 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:59.551 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:46:59 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:59.552 INFO:journalctl@ceph.osd.4.smithi184.stdout:Apr 06 02:46:59 smithi184 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-4[50341]: 2024-04-06T02:46:59.346+0000 7f705cc7e700 -1 osd.4 0 waiting for initial osdmap 2024-04-06T02:46:59.552 INFO:journalctl@ceph.osd.4.smithi184.stdout:Apr 06 02:46:59 smithi184 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-4[50341]: 2024-04-06T02:46:59.353+0000 7f705626e700 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T02:46:59.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:59 smithi105 ceph-mon[29228]: from='osd.4 [v2:172.21.15.184:6808/2237051966,v1:172.21.15.184:6809/2237051966]' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-06T02:46:59.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:59 smithi105 ceph-mon[29228]: osdmap e33: 5 total, 4 up, 5 in 2024-04-06T02:46:59.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:59 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:46:59.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:59 smithi105 ceph-mon[29228]: from='osd.4 [v2:172.21.15.184:6808/2237051966,v1:172.21.15.184:6809/2237051966]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-04-06T02:46:59.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:59 smithi105 ceph-mon[29228]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-06T02:46:59.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:59 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:46:59.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:46:59 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:00.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:00 smithi184 ceph-mon[37439]: from='osd.4 [v2:172.21.15.184:6808/2237051966,v1:172.21.15.184:6809/2237051966]' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-04-06T02:47:00.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:00 smithi184 ceph-mon[37439]: osdmap e34: 5 total, 4 up, 5 in 2024-04-06T02:47:00.480 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:00 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:47:00.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:00 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:47:00.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:00 smithi105 ceph-mon[29228]: from='osd.4 [v2:172.21.15.184:6808/2237051966,v1:172.21.15.184:6809/2237051966]' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-04-06T02:47:00.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:00 smithi105 ceph-mon[29228]: osdmap e34: 5 total, 4 up, 5 in 2024-04-06T02:47:00.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:00 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:47:00.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:00 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:47:00.833 INFO:teuthology.orchestra.run.smithi184.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-06T02:47:00.833 INFO:teuthology.orchestra.run.smithi184.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-06T02:47:00.833 INFO:teuthology.orchestra.run.smithi184.stderr: stderr: 10+0 records in 2024-04-06T02:47:00.834 INFO:teuthology.orchestra.run.smithi184.stderr:10+0 records out 2024-04-06T02:47:00.834 INFO:teuthology.orchestra.run.smithi184.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0199052 s, 527 MB/s 2024-04-06T02:47:00.834 INFO:teuthology.orchestra.run.smithi184.stderr:--> Zapping successful for: 2024-04-06T02:47:01.580 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:01 smithi184 ceph-mon[37439]: purged_snaps scrub starts 2024-04-06T02:47:01.580 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:01 smithi184 ceph-mon[37439]: purged_snaps scrub ok 2024-04-06T02:47:01.581 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:01 smithi184 ceph-mon[37439]: osd.4 [v2:172.21.15.184:6808/2237051966,v1:172.21.15.184:6809/2237051966] boot 2024-04-06T02:47:01.581 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:01 smithi184 ceph-mon[37439]: osdmap e35: 5 total, 5 up, 5 in 2024-04-06T02:47:01.581 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:01 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:47:01.581 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:01 smithi184 ceph-mon[37439]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:47:01.664 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi184:vg_nvme/lv_2 2024-04-06T02:47:01.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:01 smithi105 ceph-mon[29228]: purged_snaps scrub starts 2024-04-06T02:47:01.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:01 smithi105 ceph-mon[29228]: purged_snaps scrub ok 2024-04-06T02:47:01.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:01 smithi105 ceph-mon[29228]: osd.4 [v2:172.21.15.184:6808/2237051966,v1:172.21.15.184:6809/2237051966] boot 2024-04-06T02:47:01.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:01 smithi105 ceph-mon[29228]: osdmap e35: 5 total, 5 up, 5 in 2024-04-06T02:47:01.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:01 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T02:47:01.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:01 smithi105 ceph-mon[29228]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:47:02.401 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi184/config 2024-04-06T02:47:02.523 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:02 smithi184 ceph-mon[37439]: osdmap e36: 5 total, 5 up, 5 in 2024-04-06T02:47:02.523 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:02 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:02.523 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:02 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:02.523 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:02 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:47:02.523 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:02 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:47:02.523 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:02 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:02.523 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:02 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:47:02.524 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:02 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:47:02.524 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:02 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:02.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:02 smithi105 ceph-mon[29228]: osdmap e36: 5 total, 5 up, 5 in 2024-04-06T02:47:02.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:02 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:02.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:02 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:02.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:02 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:47:02.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:02 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:47:02.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:02 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:02.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:02 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:47:02.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:02 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:47:02.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:02 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:03.705 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:03 smithi105 ceph-mon[29228]: Detected new or changed devices on smithi184 2024-04-06T02:47:03.705 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:03 smithi105 ceph-mon[29228]: Adjusting osd_memory_target on smithi184 to 7456M 2024-04-06T02:47:03.705 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:03 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:47:03.705 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:03 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:47:03.705 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:03 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:47:03.705 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:03 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:03.705 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:03 smithi105 ceph-mon[29228]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:47:03.728 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:03 smithi184 ceph-mon[37439]: Detected new or changed devices on smithi184 2024-04-06T02:47:03.728 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:03 smithi184 ceph-mon[37439]: Adjusting osd_memory_target on smithi184 to 7456M 2024-04-06T02:47:03.729 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:03 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:47:03.729 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:03 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:47:03.729 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:03 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:47:03.729 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:03 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:03.729 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:03 smithi184 ceph-mon[37439]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:47:03.940 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.955+0000 7f890e8ae700 1 -- 172.21.15.184:0/440688288 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8908100cc0 msgr2=0x7f8908101120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:03.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.955+0000 7f890e8ae700 1 --2- 172.21.15.184:0/440688288 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8908100cc0 0x7f8908101120 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f88fc009910 tx=0x7f88fc009c20 comp rx=0 tx=0).stop 2024-04-06T02:47:03.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.955+0000 7f890e8ae700 1 -- 172.21.15.184:0/440688288 shutdown_connections 2024-04-06T02:47:03.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.955+0000 7f890e8ae700 1 --2- 172.21.15.184:0/440688288 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8908100cc0 0x7f8908101120 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:03.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.955+0000 7f890e8ae700 1 --2- 172.21.15.184:0/440688288 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89080ffac0 0x7f89080ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:03.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.955+0000 7f890e8ae700 1 -- 172.21.15.184:0/440688288 >> 172.21.15.184:0/440688288 conn(0x7f89080fb060 msgr2=0x7f89080fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:03.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.955+0000 7f890e8ae700 1 -- 172.21.15.184:0/440688288 shutdown_connections 2024-04-06T02:47:03.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.955+0000 7f890e8ae700 1 -- 172.21.15.184:0/440688288 wait complete. 2024-04-06T02:47:03.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.956+0000 7f890e8ae700 1 Processor -- start 2024-04-06T02:47:03.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.956+0000 7f890e8ae700 1 -- start start 2024-04-06T02:47:03.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.956+0000 7f890e8ae700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89080ffac0 0x7f8908198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:03.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.956+0000 7f890e8ae700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8908100cc0 0x7f8908198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:03.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.956+0000 7f890e8ae700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8908199190 con 0x7f89080ffac0 2024-04-06T02:47:03.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.956+0000 7f890e8ae700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89081992d0 con 0x7f8908100cc0 2024-04-06T02:47:03.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.956+0000 7f89077fe700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8908100cc0 0x7f8908198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:03.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.956+0000 7f89077fe700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8908100cc0 0x7f8908198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:35086/0 (socket says 172.21.15.184:35086) 2024-04-06T02:47:03.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.956+0000 7f89077fe700 1 -- 172.21.15.184:0/2761157968 learned_addr learned my addr 172.21.15.184:0/2761157968 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-04-06T02:47:03.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.956+0000 7f89077fe700 1 -- 172.21.15.184:0/2761157968 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89080ffac0 msgr2=0x7f8908198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:03.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.956+0000 7f89077fe700 1 --2- 172.21.15.184:0/2761157968 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89080ffac0 0x7f8908198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:03.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.956+0000 7f89077fe700 1 -- 172.21.15.184:0/2761157968 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88fc0095f0 con 0x7f8908100cc0 2024-04-06T02:47:03.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.957+0000 7f89077fe700 1 --2- 172.21.15.184:0/2761157968 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8908100cc0 0x7f8908198b70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f88fc006010 tx=0x7f88fc00be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:03.944 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.957+0000 7f89057fa700 1 -- 172.21.15.184:0/2761157968 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88fc01d070 con 0x7f8908100cc0 2024-04-06T02:47:03.944 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.957+0000 7f89057fa700 1 -- 172.21.15.184:0/2761157968 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f88fc00f460 con 0x7f8908100cc0 2024-04-06T02:47:03.944 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.957+0000 7f890e8ae700 1 -- 172.21.15.184:0/2761157968 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f890819dd20 con 0x7f8908100cc0 2024-04-06T02:47:03.944 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.957+0000 7f89057fa700 1 -- 172.21.15.184:0/2761157968 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88fc021640 con 0x7f8908100cc0 2024-04-06T02:47:03.944 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.957+0000 7f890e8ae700 1 -- 172.21.15.184:0/2761157968 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f890819e290 con 0x7f8908100cc0 2024-04-06T02:47:03.945 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.958+0000 7f890e8ae700 1 -- 172.21.15.184:0/2761157968 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89081086c0 con 0x7f8908100cc0 2024-04-06T02:47:03.945 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.958+0000 7f89057fa700 1 -- 172.21.15.184:0/2761157968 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f88fc004b20 con 0x7f8908100cc0 2024-04-06T02:47:03.945 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.959+0000 7f89057fa700 1 --2- 172.21.15.184:0/2761157968 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f88f006c4e0 0x7f88f006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:03.945 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.959+0000 7f89057fa700 1 -- 172.21.15.184:0/2761157968 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4206+0+0 (secure 0 0 0) 0x7f88fc08c720 con 0x7f8908100cc0 2024-04-06T02:47:03.945 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.959+0000 7f8907fff700 1 --2- 172.21.15.184:0/2761157968 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f88f006c4e0 0x7f88f006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:03.945 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.959+0000 7f8907fff700 1 --2- 172.21.15.184:0/2761157968 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f88f006c4e0 0x7f88f006e9a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8908100b20 tx=0x7f88f8006d20 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:03.949 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:03.963+0000 7f89057fa700 1 -- 172.21.15.184:0/2761157968 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f88fc05af00 con 0x7f8908100cc0 2024-04-06T02:47:04.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:04.158+0000 7f890e8ae700 1 -- 172.21.15.184:0/2761157968 --> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f89080694c0 con 0x7f88f006c4e0 2024-04-06T02:47:04.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:04 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:47:04.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:04 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:47:04.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:04 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:47:04.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:04 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T02:47:04.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:04 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T02:47:04.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:04 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:47:05.563 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:05 smithi184 ceph-mon[37439]: from='client.24159 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:47:05.563 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:05 smithi184 ceph-mon[37439]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:47:05.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:05 smithi105 ceph-mon[29228]: from='client.24159 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:47:05.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:05 smithi105 ceph-mon[29228]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:47:06.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:06 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/123189318' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95e64c1c-1f14-4017-b27c-11f447ac5a67"}]: dispatch 2024-04-06T02:47:06.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:06 smithi105 ceph-mon[29228]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95e64c1c-1f14-4017-b27c-11f447ac5a67"}]: dispatch 2024-04-06T02:47:06.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:06 smithi105 ceph-mon[29228]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "95e64c1c-1f14-4017-b27c-11f447ac5a67"}]': finished 2024-04-06T02:47:06.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:06 smithi105 ceph-mon[29228]: osdmap e37: 6 total, 5 up, 6 in 2024-04-06T02:47:06.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:06 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:47:06.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:06 smithi184 ceph-mon[37439]: from='client.? 172.21.15.184:0/123189318' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95e64c1c-1f14-4017-b27c-11f447ac5a67"}]: dispatch 2024-04-06T02:47:06.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:06 smithi184 ceph-mon[37439]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95e64c1c-1f14-4017-b27c-11f447ac5a67"}]: dispatch 2024-04-06T02:47:06.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:06 smithi184 ceph-mon[37439]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "95e64c1c-1f14-4017-b27c-11f447ac5a67"}]': finished 2024-04-06T02:47:06.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:06 smithi184 ceph-mon[37439]: osdmap e37: 6 total, 5 up, 6 in 2024-04-06T02:47:06.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:06 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:47:07.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:07 smithi184 ceph-mon[37439]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:47:07.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:07 smithi184 ceph-mon[37439]: from='client.? 172.21.15.184:0/3935784574' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:47:08.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:07 smithi105 ceph-mon[29228]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:47:08.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:07 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/3935784574' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T02:47:09.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:09 smithi184 ceph-mon[37439]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:47:09.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:09 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:47:10.059 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:09 smithi105 ceph-mon[29228]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:47:10.060 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:09 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:47:11.899 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:11 smithi184 ceph-mon[37439]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:47:11.999 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:11 smithi105 ceph-mon[29228]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:47:13.497 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:13 smithi184 ceph-mon[37439]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:47:13.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:13 smithi105 ceph-mon[29228]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:47:14.680 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:14 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-06T02:47:14.680 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:14 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:47:14.681 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:14 smithi184 ceph-mon[37439]: Deploying daemon osd.5 on smithi184 2024-04-06T02:47:14.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:14 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-06T02:47:14.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:14 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:47:14.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:14 smithi105 ceph-mon[29228]: Deploying daemon osd.5 on smithi184 2024-04-06T02:47:15.581 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:15 smithi184 ceph-mon[37439]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:47:15.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:15 smithi105 ceph-mon[29228]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:47:17.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:17 smithi105 ceph-mon[29228]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:47:17.958 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:17 smithi184 ceph-mon[37439]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:47:19.831 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:19 smithi184 ceph-mon[37439]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:47:19.831 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:19 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:19.831 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:19 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:19.831 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:19 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:47:19.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:19 smithi105 ceph-mon[29228]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:47:19.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:19 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:19.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:19 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:19.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:19 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:47:21.793 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:21 smithi184 ceph-mon[37439]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:47:21.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:21 smithi105 ceph-mon[29228]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:47:22.558 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:22.572+0000 7f89057fa700 1 -- 172.21.15.184:0/2761157968 <== mgr.14221 v2:172.21.15.105:6800/2334550011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f89080694c0 con 0x7f88f006c4e0 2024-04-06T02:47:22.558 INFO:teuthology.orchestra.run.smithi184.stdout:Created osd(s) 5 on host 'smithi184' 2024-04-06T02:47:22.560 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:22.574+0000 7f890e8ae700 1 -- 172.21.15.184:0/2761157968 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f88f006c4e0 msgr2=0x7f88f006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:22.560 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:22.574+0000 7f890e8ae700 1 --2- 172.21.15.184:0/2761157968 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f88f006c4e0 0x7f88f006e9a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8908100b20 tx=0x7f88f8006d20 comp rx=0 tx=0).stop 2024-04-06T02:47:22.560 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:22.574+0000 7f890e8ae700 1 -- 172.21.15.184:0/2761157968 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8908100cc0 msgr2=0x7f8908198b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:22.560 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:22.574+0000 7f890e8ae700 1 --2- 172.21.15.184:0/2761157968 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8908100cc0 0x7f8908198b70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f88fc006010 tx=0x7f88fc00be20 comp rx=0 tx=0).stop 2024-04-06T02:47:22.560 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:22.574+0000 7f890e8ae700 1 -- 172.21.15.184:0/2761157968 shutdown_connections 2024-04-06T02:47:22.560 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:22.574+0000 7f890e8ae700 1 --2- 172.21.15.184:0/2761157968 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f88f006c4e0 0x7f88f006e9a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:22.560 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:22.574+0000 7f890e8ae700 1 --2- 172.21.15.184:0/2761157968 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89080ffac0 0x7f8908198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:22.560 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:22.574+0000 7f890e8ae700 1 --2- 172.21.15.184:0/2761157968 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8908100cc0 0x7f8908198b70 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:22.560 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:22.574+0000 7f890e8ae700 1 -- 172.21.15.184:0/2761157968 >> 172.21.15.184:0/2761157968 conn(0x7f89080fb060 msgr2=0x7f8908068db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:22.560 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:22.574+0000 7f890e8ae700 1 -- 172.21.15.184:0/2761157968 shutdown_connections 2024-04-06T02:47:22.561 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:22.574+0000 7f890e8ae700 1 -- 172.21.15.184:0/2761157968 wait complete. 2024-04-06T02:47:23.524 DEBUG:teuthology.orchestra.run.smithi184:osd.5> sudo journalctl -f -n 0 -u ceph-130df578-f3bf-11ee-b649-cb9ed24678a4@osd.5.service 2024-04-06T02:47:23.526 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-04-06T02:47:23.526 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph osd stat -f json 2024-04-06T02:47:23.748 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:47:23.870 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:23 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:23.870 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:23 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:23.870 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:23 smithi105 ceph-mon[29228]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:47:23.870 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:23 smithi105 ceph-mon[29228]: from='osd.5 [v2:172.21.15.184:6816/1371998009,v1:172.21.15.184:6817/1371998009]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T02:47:23.870 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:23 smithi105 ceph-mon[29228]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T02:47:23.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:23 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:23.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:23 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:23.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:23 smithi184 ceph-mon[37439]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:47:23.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:23 smithi184 ceph-mon[37439]: from='osd.5 [v2:172.21.15.184:6816/1371998009,v1:172.21.15.184:6817/1371998009]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T02:47:23.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:23 smithi184 ceph-mon[37439]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T02:47:24.882 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:24 smithi184 ceph-mon[37439]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-06T02:47:24.883 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:24 smithi184 ceph-mon[37439]: osdmap e38: 6 total, 5 up, 6 in 2024-04-06T02:47:24.883 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:24 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:47:24.883 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:24 smithi184 ceph-mon[37439]: from='osd.5 [v2:172.21.15.184:6816/1371998009,v1:172.21.15.184:6817/1371998009]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-04-06T02:47:24.883 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:24 smithi184 ceph-mon[37439]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-04-06T02:47:24.883 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:24 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:24.883 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:24 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:24.883 INFO:journalctl@ceph.osd.5.smithi184.stdout:Apr 06 02:47:24 smithi184 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-5[57293]: 2024-04-06T02:47:24.578+0000 7fbb6d5d5700 -1 osd.5 0 waiting for initial osdmap 2024-04-06T02:47:24.883 INFO:journalctl@ceph.osd.5.smithi184.stdout:Apr 06 02:47:24 smithi184 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-5[57293]: 2024-04-06T02:47:24.586+0000 7fbb69bcb700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T02:47:24.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:24 smithi105 ceph-mon[29228]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-06T02:47:24.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:24 smithi105 ceph-mon[29228]: osdmap e38: 6 total, 5 up, 6 in 2024-04-06T02:47:24.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:24 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:47:24.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:24 smithi105 ceph-mon[29228]: from='osd.5 [v2:172.21.15.184:6816/1371998009,v1:172.21.15.184:6817/1371998009]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-04-06T02:47:24.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:24 smithi105 ceph-mon[29228]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-04-06T02:47:24.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:24 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:24.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:24 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:25.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.547+0000 7f3f6a5bb700 1 -- 172.21.15.105:0/276438397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3f64100cc0 msgr2=0x7f3f64101120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:25.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.547+0000 7f3f6a5bb700 1 --2- 172.21.15.105:0/276438397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3f64100cc0 0x7f3f64101120 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009910 tx=0x7f3f54009c20 comp rx=0 tx=0).stop 2024-04-06T02:47:25.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.547+0000 7f3f6a5bb700 1 -- 172.21.15.105:0/276438397 shutdown_connections 2024-04-06T02:47:25.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.547+0000 7f3f6a5bb700 1 --2- 172.21.15.105:0/276438397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3f64100cc0 0x7f3f64101120 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:25.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.547+0000 7f3f6a5bb700 1 --2- 172.21.15.105:0/276438397 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f640ffac0 0x7f3f640ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:25.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.547+0000 7f3f6a5bb700 1 -- 172.21.15.105:0/276438397 >> 172.21.15.105:0/276438397 conn(0x7f3f640fb060 msgr2=0x7f3f640fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:25.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.547+0000 7f3f6a5bb700 1 -- 172.21.15.105:0/276438397 shutdown_connections 2024-04-06T02:47:25.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.547+0000 7f3f6a5bb700 1 -- 172.21.15.105:0/276438397 wait complete. 2024-04-06T02:47:25.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.548+0000 7f3f6a5bb700 1 Processor -- start 2024-04-06T02:47:25.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.548+0000 7f3f6a5bb700 1 -- start start 2024-04-06T02:47:25.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.548+0000 7f3f6a5bb700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f640ffac0 0x7f3f64069970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:25.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.548+0000 7f3f6a5bb700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3f64100cc0 0x7f3f6419b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:25.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.548+0000 7f3f6a5bb700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6419b690 con 0x7f3f64100cc0 2024-04-06T02:47:25.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.548+0000 7f3f6a5bb700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6419b7d0 con 0x7f3f640ffac0 2024-04-06T02:47:25.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.548+0000 7f3f637fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3f64100cc0 0x7f3f6419b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:25.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.548+0000 7f3f637fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3f64100cc0 0x7f3f6419b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:48128/0 (socket says 172.21.15.105:48128) 2024-04-06T02:47:25.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.548+0000 7f3f637fe700 1 -- 172.21.15.105:0/3121988109 learned_addr learned my addr 172.21.15.105:0/3121988109 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:47:25.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.548+0000 7f3f63fff700 1 --2- 172.21.15.105:0/3121988109 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f640ffac0 0x7f3f64069970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:25.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.548+0000 7f3f637fe700 1 -- 172.21.15.105:0/3121988109 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f640ffac0 msgr2=0x7f3f64069970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:25.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.548+0000 7f3f637fe700 1 --2- 172.21.15.105:0/3121988109 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f640ffac0 0x7f3f64069970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:25.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.549+0000 7f3f637fe700 1 -- 172.21.15.105:0/3121988109 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f540095f0 con 0x7f3f64100cc0 2024-04-06T02:47:25.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.549+0000 7f3f637fe700 1 --2- 172.21.15.105:0/3121988109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3f64100cc0 0x7f3f6419b070 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400bd70 tx=0x7f3f5400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:25.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.549+0000 7f3f617fa700 1 -- 172.21.15.105:0/3121988109 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f5401d070 con 0x7f3f64100cc0 2024-04-06T02:47:25.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.549+0000 7f3f617fa700 1 -- 172.21.15.105:0/3121988109 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3f54022470 con 0x7f3f64100cc0 2024-04-06T02:47:25.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.549+0000 7f3f6a5bb700 1 -- 172.21.15.105:0/3121988109 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f6419ba50 con 0x7f3f64100cc0 2024-04-06T02:47:25.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.549+0000 7f3f617fa700 1 -- 172.21.15.105:0/3121988109 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f5400f670 con 0x7f3f64100cc0 2024-04-06T02:47:25.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.549+0000 7f3f6a5bb700 1 -- 172.21.15.105:0/3121988109 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f6419bf40 con 0x7f3f64100cc0 2024-04-06T02:47:25.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.550+0000 7f3f6a5bb700 1 -- 172.21.15.105:0/3121988109 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f64108620 con 0x7f3f64100cc0 2024-04-06T02:47:25.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.550+0000 7f3f617fa700 1 -- 172.21.15.105:0/3121988109 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f3f54004b70 con 0x7f3f64100cc0 2024-04-06T02:47:25.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.551+0000 7f3f617fa700 1 --2- 172.21.15.105:0/3121988109 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f3f4c06c4e0 0x7f3f4c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:25.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.551+0000 7f3f617fa700 1 -- 172.21.15.105:0/3121988109 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4358+0+0 (secure 0 0 0) 0x7f3f5408cad0 con 0x7f3f64100cc0 2024-04-06T02:47:25.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.551+0000 7f3f63fff700 1 --2- 172.21.15.105:0/3121988109 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f3f4c06c4e0 0x7f3f4c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:25.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.551+0000 7f3f63fff700 1 --2- 172.21.15.105:0/3121988109 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f3f4c06c4e0 0x7f3f4c06e9a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3f58005fd0 tx=0x7f3f58005dd0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:25.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.555+0000 7f3f617fa700 1 -- 172.21.15.105:0/3121988109 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3f5405b240 con 0x7f3f64100cc0 2024-04-06T02:47:25.699 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:25 smithi184 ceph-mon[37439]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-04-06T02:47:25.699 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:25 smithi184 ceph-mon[37439]: osdmap e39: 6 total, 5 up, 6 in 2024-04-06T02:47:25.700 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:25 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:47:25.700 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:25 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:47:25.700 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:25 smithi184 ceph-mon[37439]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:47:25.700 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:25 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:47:25.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.745+0000 7f3f6a5bb700 1 -- 172.21.15.105:0/3121988109 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f3f6419c610 con 0x7f3f64100cc0 2024-04-06T02:47:25.731 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:25 smithi105 ceph-mon[29228]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-04-06T02:47:25.731 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:25 smithi105 ceph-mon[29228]: osdmap e39: 6 total, 5 up, 6 in 2024-04-06T02:47:25.731 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:25 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:47:25.731 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:25 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:47:25.731 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:25 smithi105 ceph-mon[29228]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-06T02:47:25.732 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:25 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:47:25.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.746+0000 7f3f617fa700 1 -- 172.21.15.105:0/3121988109 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7f3f5405add0 con 0x7f3f64100cc0 2024-04-06T02:47:25.732 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:47:25.734 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.749+0000 7f3f6a5bb700 1 -- 172.21.15.105:0/3121988109 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f3f4c06c4e0 msgr2=0x7f3f4c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:25.734 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.749+0000 7f3f6a5bb700 1 --2- 172.21.15.105:0/3121988109 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f3f4c06c4e0 0x7f3f4c06e9a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3f58005fd0 tx=0x7f3f58005dd0 comp rx=0 tx=0).stop 2024-04-06T02:47:25.734 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.749+0000 7f3f6a5bb700 1 -- 172.21.15.105:0/3121988109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3f64100cc0 msgr2=0x7f3f6419b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:25.735 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.749+0000 7f3f6a5bb700 1 --2- 172.21.15.105:0/3121988109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3f64100cc0 0x7f3f6419b070 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400bd70 tx=0x7f3f5400be50 comp rx=0 tx=0).stop 2024-04-06T02:47:25.735 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.749+0000 7f3f6a5bb700 1 -- 172.21.15.105:0/3121988109 shutdown_connections 2024-04-06T02:47:25.735 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.749+0000 7f3f6a5bb700 1 --2- 172.21.15.105:0/3121988109 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f3f4c06c4e0 0x7f3f4c06e9a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:25.735 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.749+0000 7f3f6a5bb700 1 --2- 172.21.15.105:0/3121988109 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f640ffac0 0x7f3f64069970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:25.735 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.749+0000 7f3f6a5bb700 1 --2- 172.21.15.105:0/3121988109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3f64100cc0 0x7f3f6419b070 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:25.735 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.749+0000 7f3f6a5bb700 1 -- 172.21.15.105:0/3121988109 >> 172.21.15.105:0/3121988109 conn(0x7f3f640fb060 msgr2=0x7f3f64103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:25.735 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.749+0000 7f3f6a5bb700 1 -- 172.21.15.105:0/3121988109 shutdown_connections 2024-04-06T02:47:25.735 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:25.749+0000 7f3f6a5bb700 1 -- 172.21.15.105:0/3121988109 wait complete. 2024-04-06T02:47:26.518 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1712371645,"num_in_osds":6,"osd_in_since":1712371626,"num_remapped_pgs":0} 2024-04-06T02:47:26.519 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-06T02:47:26.728 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:47:26.795 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:26 smithi105 ceph-mon[29228]: purged_snaps scrub starts 2024-04-06T02:47:26.796 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:26 smithi105 ceph-mon[29228]: purged_snaps scrub ok 2024-04-06T02:47:26.796 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:26 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:47:26.796 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:26 smithi105 ceph-mon[29228]: osd.5 [v2:172.21.15.184:6816/1371998009,v1:172.21.15.184:6817/1371998009] boot 2024-04-06T02:47:26.796 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:26 smithi105 ceph-mon[29228]: osdmap e40: 6 total, 6 up, 6 in 2024-04-06T02:47:26.796 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:26 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:47:26.796 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:26 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/3121988109' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-06T02:47:26.886 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:26 smithi184 ceph-mon[37439]: purged_snaps scrub starts 2024-04-06T02:47:26.886 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:26 smithi184 ceph-mon[37439]: purged_snaps scrub ok 2024-04-06T02:47:26.886 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:26 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:47:26.886 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:26 smithi184 ceph-mon[37439]: osd.5 [v2:172.21.15.184:6816/1371998009,v1:172.21.15.184:6817/1371998009] boot 2024-04-06T02:47:26.886 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:26 smithi184 ceph-mon[37439]: osdmap e40: 6 total, 6 up, 6 in 2024-04-06T02:47:26.886 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:26 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T02:47:26.886 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:26 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/3121988109' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-06T02:47:27.871 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:27 smithi105 ceph-mon[29228]: osdmap e41: 6 total, 6 up, 6 in 2024-04-06T02:47:27.871 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:27 smithi105 ceph-mon[29228]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:27.871 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:27.871 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:27.871 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:47:27.871 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:47:27.871 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:47:27.871 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:27.871 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:47:27.871 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:47:27.871 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:27.872 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:27 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:47:27.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:27 smithi184 ceph-mon[37439]: osdmap e41: 6 total, 6 up, 6 in 2024-04-06T02:47:27.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:27 smithi184 ceph-mon[37439]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:27.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:27.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:27.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:47:27.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:47:27.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T02:47:27.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:27.982 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:47:27.982 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:47:27.982 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:27.982 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:27 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:47:28.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.463+0000 7fafd5841700 1 -- 172.21.15.105:0/1355621782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fafd0101780 msgr2=0x7fafd01057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:28.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.463+0000 7fafd5841700 1 --2- 172.21.15.105:0/1355621782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fafd0101780 0x7fafd01057d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fafc4009910 tx=0x7fafc4009c20 comp rx=0 tx=0).stop 2024-04-06T02:47:28.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.464+0000 7fafd5841700 1 -- 172.21.15.105:0/1355621782 shutdown_connections 2024-04-06T02:47:28.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.464+0000 7fafd5841700 1 --2- 172.21.15.105:0/1355621782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fafd0101780 0x7fafd01057d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:28.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.464+0000 7fafd5841700 1 --2- 172.21.15.105:0/1355621782 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fafd0100dd0 0x7fafd01011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:28.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.464+0000 7fafd5841700 1 -- 172.21.15.105:0/1355621782 >> 172.21.15.105:0/1355621782 conn(0x7fafd00fc640 msgr2=0x7fafd00fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:28.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.464+0000 7fafd5841700 1 -- 172.21.15.105:0/1355621782 shutdown_connections 2024-04-06T02:47:28.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.464+0000 7fafd5841700 1 -- 172.21.15.105:0/1355621782 wait complete. 2024-04-06T02:47:28.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.464+0000 7fafd5841700 1 Processor -- start 2024-04-06T02:47:28.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.464+0000 7fafd5841700 1 -- start start 2024-04-06T02:47:28.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.465+0000 7fafd5841700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fafd0100dd0 0x7fafd019e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:28.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.465+0000 7fafd5841700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fafd0101780 0x7fafd019ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:28.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.465+0000 7fafd5841700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd019f400 con 0x7fafd0100dd0 2024-04-06T02:47:28.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.465+0000 7fafd5841700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd01988b0 con 0x7fafd0101780 2024-04-06T02:47:28.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.465+0000 7fafceffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fafd0100dd0 0x7fafd019e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:28.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.465+0000 7fafceffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fafd0100dd0 0x7fafd019e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:48146/0 (socket says 172.21.15.105:48146) 2024-04-06T02:47:28.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.465+0000 7fafceffd700 1 -- 172.21.15.105:0/91335167 learned_addr learned my addr 172.21.15.105:0/91335167 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:47:28.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.465+0000 7fafce7fc700 1 --2- 172.21.15.105:0/91335167 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fafd0101780 0x7fafd019ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:28.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.465+0000 7fafceffd700 1 -- 172.21.15.105:0/91335167 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fafd0101780 msgr2=0x7fafd019ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:28.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.465+0000 7fafceffd700 1 --2- 172.21.15.105:0/91335167 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fafd0101780 0x7fafd019ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:28.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.465+0000 7fafceffd700 1 -- 172.21.15.105:0/91335167 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc40095f0 con 0x7fafd0100dd0 2024-04-06T02:47:28.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.465+0000 7fafce7fc700 1 --2- 172.21.15.105:0/91335167 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fafd0101780 0x7fafd019ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T02:47:28.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.465+0000 7fafceffd700 1 --2- 172.21.15.105:0/91335167 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fafd0100dd0 0x7fafd019e830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fafc000b4b0 tx=0x7fafc000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:28.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.465+0000 7fafd483f700 1 -- 172.21.15.105:0/91335167 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafc0011800 con 0x7fafd0100dd0 2024-04-06T02:47:28.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.465+0000 7fafd5841700 1 -- 172.21.15.105:0/91335167 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd0198b90 con 0x7fafd0100dd0 2024-04-06T02:47:28.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.465+0000 7fafd483f700 1 -- 172.21.15.105:0/91335167 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fafc0011e10 con 0x7fafd0100dd0 2024-04-06T02:47:28.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.465+0000 7fafd483f700 1 -- 172.21.15.105:0/91335167 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafc000f2e0 con 0x7fafd0100dd0 2024-04-06T02:47:28.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.465+0000 7fafd5841700 1 -- 172.21.15.105:0/91335167 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd01990e0 con 0x7fafd0100dd0 2024-04-06T02:47:28.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.467+0000 7fafd5841700 1 -- 172.21.15.105:0/91335167 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd0104f50 con 0x7fafd0100dd0 2024-04-06T02:47:28.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.467+0000 7fafd483f700 1 -- 172.21.15.105:0/91335167 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fafc000f440 con 0x7fafd0100dd0 2024-04-06T02:47:28.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.467+0000 7fafd483f700 1 --2- 172.21.15.105:0/91335167 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fafb806c4e0 0x7fafb806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:28.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.467+0000 7fafce7fc700 1 --2- 172.21.15.105:0/91335167 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fafb806c4e0 0x7fafb806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:28.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.467+0000 7fafd483f700 1 -- 172.21.15.105:0/91335167 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fafc008b050 con 0x7fafd0100dd0 2024-04-06T02:47:28.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.468+0000 7fafce7fc700 1 --2- 172.21.15.105:0/91335167 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fafb806c4e0 0x7fafb806e9a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fafc4006010 tx=0x7fafc4005960 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:28.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.472+0000 7fafd483f700 1 -- 172.21.15.105:0/91335167 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fafc0059680 con 0x7fafd0100dd0 2024-04-06T02:47:28.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.665+0000 7fafd5841700 1 -- 172.21.15.105:0/91335167 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fafd01993b0 con 0x7fafd0100dd0 2024-04-06T02:47:28.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.666+0000 7fafd483f700 1 -- 172.21.15.105:0/91335167 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11389 (secure 0 0 0) 0x7fafc0059210 con 0x7fafd0100dd0 2024-04-06T02:47:28.652 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:47:28.652 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":41,"fsid":"130df578-f3bf-11ee-b649-cb9ed24678a4","created":"2024-04-06T02:41:32.644992+0000","modified":"2024-04-06T02:47:26.582254+0000","last_up_change":"2024-04-06T02:47:25.576978+0000","last_in_change":"2024-04-06T02:47:06.634446+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T02:46:12.900794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41c20c92-5314-4354-b659-cb449ef98c81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2018109658},{"type":"v1","addr":"172.21.15.105:6803","nonce":2018109658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2018109658},{"type":"v1","addr":"172.21.15.105:6805","nonce":2018109658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":2018109658},{"type":"v1","addr":"172.21.15.105:6809","nonce":2018109658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2018109658},{"type":"v1","addr":"172.21.15.105:6807","nonce":2018109658}]},"public_addr":"172.21.15.105:6803/2018109658","cluster_addr":"172.21.15.105:6805/2018109658","heartbeat_back_addr":"172.21.15.105:6809/2018109658","heartbeat_front_addr":"172.21.15.105:6807/2018109658","state":["exists","up"]},{"osd":1,"uuid":"ea3f1b19-62f8-488e-b62a-2340aef1112a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":1345191629},{"type":"v1","addr":"172.21.15.105:6811","nonce":1345191629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":1345191629},{"type":"v1","addr":"172.21.15.105:6813","nonce":1345191629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":1345191629},{"type":"v1","addr":"172.21.15.105:6817","nonce":1345191629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":1345191629},{"type":"v1","addr":"172.21.15.105:6815","nonce":1345191629}]},"public_addr":"172.21.15.105:6811/1345191629","cluster_addr":"172.21.15.105:6813/1345191629","heartbeat_back_addr":"172.21.15.105:6817/1345191629","heartbeat_front_addr":"172.21.15.105:6815/1345191629","state":["exists","up"]},{"osd":2,"uuid":"def7693a-bb30-4c74-ae20-59a7a86ef9db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":3651690266},{"type":"v1","addr":"172.21.15.105:6819","nonce":3651690266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":3651690266},{"type":"v1","addr":"172.21.15.105:6821","nonce":3651690266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3651690266},{"type":"v1","addr":"172.21.15.105:6825","nonce":3651690266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":3651690266},{"type":"v1","addr":"172.21.15.105:6823","nonce":3651690266}]},"public_addr":"172.21.15.105:6819/3651690266","cluster_addr":"172.21.15.105:6821/3651690266","heartbeat_back_addr":"172.21.15.105:6825/3651690266","heartbeat_front_addr":"172.21.15.105:6823/3651690266","state":["exists","up"]},{"osd":3,"uuid":"97f959d7-5922-4423-9d63-e2fe677a7019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":330017535},{"type":"v1","addr":"172.21.15.184:6801","nonce":330017535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":330017535},{"type":"v1","addr":"172.21.15.184:6803","nonce":330017535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":330017535},{"type":"v1","addr":"172.21.15.184:6807","nonce":330017535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":330017535},{"type":"v1","addr":"172.21.15.184:6805","nonce":330017535}]},"public_addr":"172.21.15.184:6801/330017535","cluster_addr":"172.21.15.184:6803/330017535","heartbeat_back_addr":"172.21.15.184:6807/330017535","heartbeat_front_addr":"172.21.15.184:6805/330017535","state":["exists","up"]},{"osd":4,"uuid":"34d7ebbd-c725-4948-ae33-3ffb40ec9979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2237051966},{"type":"v1","addr":"172.21.15.184:6809","nonce":2237051966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2237051966},{"type":"v1","addr":"172.21.15.184:6811","nonce":2237051966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2237051966},{"type":"v1","addr":"172.21.15.184:6815","nonce":2237051966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2237051966},{"type":"v1","addr":"172.21.15.184:6813","nonce":2237051966}]},"public_addr":"172.21.15.184:6809/2237051966","cluster_addr":"172.21.15.184:6811/2237051966","heartbeat_back_addr":"172.21.15.184:6815/2237051966","heartbeat_front_addr":"172.21.15.184:6813/2237051966","state":["exists","up"]},{"osd":5,"uuid":"95e64c1c-1f14-4017-b27c-11f447ac5a67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":1371998009},{"type":"v1","addr":"172.21.15.184:6817","nonce":1371998009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":1371998009},{"type":"v1","addr":"172.21.15.184:6819","nonce":1371998009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":1371998009},{"type":"v1","addr":"172.21.15.184:6823","nonce":1371998009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":1371998009},{"type":"v1","addr":"172.21.15.184:6821","nonce":1371998009}]},"public_addr":"172.21.15.184:6817/1371998009","cluster_addr":"172.21.15.184:6819/1371998009","heartbeat_back_addr":"172.21.15.184:6823/1371998009","heartbeat_front_addr":"172.21.15.184:6821/1371998009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:45:12.515625+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:45:41.117291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:46:09.180416+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:46:33.131470+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:46:58.464606+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:47:24.027273+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:0/2231570326":"2024-04-07T02:43:54.575029+0000","172.21.15.105:0/1342777414":"2024-04-07T02:43:54.575029+0000","172.21.15.105:6800/1949998385":"2024-04-07T02:43:54.575029+0000","172.21.15.105:6801/1394104199":"2024-04-07T02:42:35.032003+0000","172.21.15.105:6800/1394104199":"2024-04-07T02:42:35.032003+0000","172.21.15.105:0/2131480441":"2024-04-07T02:42:35.032003+0000","172.21.15.105:0/1539671918":"2024-04-07T02:42:02.154709+0000","172.21.15.105:0/1826925654":"2024-04-07T02:42:02.154709+0000","172.21.15.105:6801/3245193613":"2024-04-07T02:42:02.154709+0000","172.21.15.105:0/403675292":"2024-04-07T02:43:54.575029+0000","172.21.15.105:6800/3245193613":"2024-04-07T02:42:02.154709+0000","172.21.15.105:0/1323549385":"2024-04-07T02:42:02.154709+0000","172.21.15.105:6801/1949998385":"2024-04-07T02:43:54.575029+0000","172.21.15.105:0/1155423922":"2024-04-07T02:42:35.032003+0000","172.21.15.105:0/3833744400":"2024-04-07T02:42:35.032003+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T02:47:28.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.668+0000 7fafd5841700 1 -- 172.21.15.105:0/91335167 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fafb806c4e0 msgr2=0x7fafb806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:28.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.668+0000 7fafd5841700 1 --2- 172.21.15.105:0/91335167 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fafb806c4e0 0x7fafb806e9a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fafc4006010 tx=0x7fafc4005960 comp rx=0 tx=0).stop 2024-04-06T02:47:28.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.668+0000 7fafd5841700 1 -- 172.21.15.105:0/91335167 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fafd0100dd0 msgr2=0x7fafd019e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:28.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.668+0000 7fafd5841700 1 --2- 172.21.15.105:0/91335167 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fafd0100dd0 0x7fafd019e830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fafc000b4b0 tx=0x7fafc000b870 comp rx=0 tx=0).stop 2024-04-06T02:47:28.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.668+0000 7fafd5841700 1 -- 172.21.15.105:0/91335167 shutdown_connections 2024-04-06T02:47:28.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.668+0000 7fafd5841700 1 --2- 172.21.15.105:0/91335167 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fafb806c4e0 0x7fafb806e9a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:28.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.668+0000 7fafd5841700 1 --2- 172.21.15.105:0/91335167 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fafd0100dd0 0x7fafd019e830 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:28.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.668+0000 7fafd5841700 1 --2- 172.21.15.105:0/91335167 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fafd0101780 0x7fafd019ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:28.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.668+0000 7fafd5841700 1 -- 172.21.15.105:0/91335167 >> 172.21.15.105:0/91335167 conn(0x7fafd00fc640 msgr2=0x7fafd00fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:28.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.668+0000 7fafd5841700 1 -- 172.21.15.105:0/91335167 shutdown_connections 2024-04-06T02:47:28.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:28.668+0000 7fafd5841700 1 -- 172.21.15.105:0/91335167 wait complete. 2024-04-06T02:47:28.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:28 smithi105 ceph-mon[29228]: Detected new or changed devices on smithi184 2024-04-06T02:47:28.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:28 smithi105 ceph-mon[29228]: Adjusting osd_memory_target on smithi184 to 4970M 2024-04-06T02:47:28.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:28 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:47:28.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:28 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:47:28.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:28 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:28.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:28 smithi184 ceph-mon[37439]: Detected new or changed devices on smithi184 2024-04-06T02:47:28.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:28 smithi184 ceph-mon[37439]: Adjusting osd_memory_target on smithi184 to 4970M 2024-04-06T02:47:28.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:28 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:47:28.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:28 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:47:28.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:28 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:47:29.396 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T02:46:12.900794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T02:47:29.397 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph osd pool get .mgr pg_num 2024-04-06T02:47:29.608 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:47:29.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:29 smithi105 ceph-mon[29228]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:29.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:29 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/91335167' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T02:47:29.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:29 smithi184 ceph-mon[37439]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:29.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:29 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/91335167' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T02:47:31.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.122+0000 7f9a74364700 1 -- 172.21.15.105:0/3056718252 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a6c105a40 msgr2=0x7f9a6c105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:31.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.122+0000 7f9a74364700 1 --2- 172.21.15.105:0/3056718252 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a6c105a40 0x7f9a6c105e20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9a68009910 tx=0x7f9a68009c20 comp rx=0 tx=0).stop 2024-04-06T02:47:31.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.122+0000 7f9a74364700 1 -- 172.21.15.105:0/3056718252 shutdown_connections 2024-04-06T02:47:31.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.122+0000 7f9a74364700 1 --2- 172.21.15.105:0/3056718252 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9a6c0ffa40 0x7f9a6c0ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:31.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.122+0000 7f9a74364700 1 --2- 172.21.15.105:0/3056718252 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a6c105a40 0x7f9a6c105e20 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:31.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.122+0000 7f9a74364700 1 -- 172.21.15.105:0/3056718252 >> 172.21.15.105:0/3056718252 conn(0x7f9a6c0fb550 msgr2=0x7f9a6c0fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:31.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.122+0000 7f9a74364700 1 -- 172.21.15.105:0/3056718252 shutdown_connections 2024-04-06T02:47:31.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.122+0000 7f9a74364700 1 -- 172.21.15.105:0/3056718252 wait complete. 2024-04-06T02:47:31.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.123+0000 7f9a74364700 1 Processor -- start 2024-04-06T02:47:31.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.123+0000 7f9a74364700 1 -- start start 2024-04-06T02:47:31.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.123+0000 7f9a74364700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9a6c0ffa40 0x7f9a6c19c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:31.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.123+0000 7f9a74364700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a6c105a40 0x7f9a6c19cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:31.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.123+0000 7f9a74364700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c19d150 con 0x7f9a6c105a40 2024-04-06T02:47:31.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.123+0000 7f9a74364700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c196600 con 0x7f9a6c0ffa40 2024-04-06T02:47:31.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.123+0000 7f9a718ff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a6c105a40 0x7f9a6c19cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:31.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.123+0000 7f9a718ff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a6c105a40 0x7f9a6c19cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:58374/0 (socket says 172.21.15.105:58374) 2024-04-06T02:47:31.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.123+0000 7f9a718ff700 1 -- 172.21.15.105:0/1564704024 learned_addr learned my addr 172.21.15.105:0/1564704024 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:47:31.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.123+0000 7f9a72100700 1 --2- 172.21.15.105:0/1564704024 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9a6c0ffa40 0x7f9a6c19c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:31.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.123+0000 7f9a718ff700 1 -- 172.21.15.105:0/1564704024 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9a6c0ffa40 msgr2=0x7f9a6c19c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:31.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.123+0000 7f9a718ff700 1 --2- 172.21.15.105:0/1564704024 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9a6c0ffa40 0x7f9a6c19c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:31.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.123+0000 7f9a718ff700 1 -- 172.21.15.105:0/1564704024 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a680095f0 con 0x7f9a6c105a40 2024-04-06T02:47:31.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.123+0000 7f9a72100700 1 --2- 172.21.15.105:0/1564704024 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9a6c0ffa40 0x7f9a6c19c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T02:47:31.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.123+0000 7f9a718ff700 1 --2- 172.21.15.105:0/1564704024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a6c105a40 0x7f9a6c19cac0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c00d8d0 tx=0x7f9a5c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:31.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.124+0000 7f9a637fe700 1 -- 172.21.15.105:0/1564704024 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a5c009940 con 0x7f9a6c105a40 2024-04-06T02:47:31.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.124+0000 7f9a637fe700 1 -- 172.21.15.105:0/1564704024 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9a5c010460 con 0x7f9a6c105a40 2024-04-06T02:47:31.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.124+0000 7f9a74364700 1 -- 172.21.15.105:0/1564704024 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a6c1968e0 con 0x7f9a6c105a40 2024-04-06T02:47:31.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.124+0000 7f9a637fe700 1 -- 172.21.15.105:0/1564704024 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a5c00f5f0 con 0x7f9a6c105a40 2024-04-06T02:47:31.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.124+0000 7f9a74364700 1 -- 172.21.15.105:0/1564704024 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a6c196e30 con 0x7f9a6c105a40 2024-04-06T02:47:31.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.125+0000 7f9a74364700 1 -- 172.21.15.105:0/1564704024 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a6c109050 con 0x7f9a6c105a40 2024-04-06T02:47:31.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.125+0000 7f9a637fe700 1 -- 172.21.15.105:0/1564704024 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f9a5c0105d0 con 0x7f9a6c105a40 2024-04-06T02:47:31.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.125+0000 7f9a637fe700 1 --2- 172.21.15.105:0/1564704024 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f9a5806c4e0 0x7f9a5806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:31.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.125+0000 7f9a637fe700 1 -- 172.21.15.105:0/1564704024 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f9a5c08b3f0 con 0x7f9a6c105a40 2024-04-06T02:47:31.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.125+0000 7f9a72100700 1 --2- 172.21.15.105:0/1564704024 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f9a5806c4e0 0x7f9a5806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:31.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.126+0000 7f9a72100700 1 --2- 172.21.15.105:0/1564704024 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f9a5806c4e0 0x7f9a5806e9a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9a6800b5c0 tx=0x7f9a6801a040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:31.115 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.130+0000 7f9a637fe700 1 -- 172.21.15.105:0/1564704024 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f9a5c059a20 con 0x7f9a6c105a40 2024-04-06T02:47:31.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.318+0000 7f9a74364700 1 -- 172.21.15.105:0/1564704024 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9a6c1970a0 con 0x7f9a6c105a40 2024-04-06T02:47:31.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.319+0000 7f9a637fe700 1 -- 172.21.15.105:0/1564704024 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9a5c08b070 con 0x7f9a6c105a40 2024-04-06T02:47:31.304 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 1 2024-04-06T02:47:31.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.321+0000 7f9a74364700 1 -- 172.21.15.105:0/1564704024 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f9a5806c4e0 msgr2=0x7f9a5806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:31.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.321+0000 7f9a74364700 1 --2- 172.21.15.105:0/1564704024 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f9a5806c4e0 0x7f9a5806e9a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9a6800b5c0 tx=0x7f9a6801a040 comp rx=0 tx=0).stop 2024-04-06T02:47:31.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.321+0000 7f9a74364700 1 -- 172.21.15.105:0/1564704024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a6c105a40 msgr2=0x7f9a6c19cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:31.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.321+0000 7f9a74364700 1 --2- 172.21.15.105:0/1564704024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a6c105a40 0x7f9a6c19cac0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c00d8d0 tx=0x7f9a5c00dc90 comp rx=0 tx=0).stop 2024-04-06T02:47:31.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.321+0000 7f9a74364700 1 -- 172.21.15.105:0/1564704024 shutdown_connections 2024-04-06T02:47:31.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.321+0000 7f9a74364700 1 --2- 172.21.15.105:0/1564704024 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f9a5806c4e0 0x7f9a5806e9a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:31.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.321+0000 7f9a74364700 1 --2- 172.21.15.105:0/1564704024 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9a6c0ffa40 0x7f9a6c19c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:31.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.321+0000 7f9a74364700 1 --2- 172.21.15.105:0/1564704024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a6c105a40 0x7f9a6c19cac0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:31.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.321+0000 7f9a74364700 1 -- 172.21.15.105:0/1564704024 >> 172.21.15.105:0/1564704024 conn(0x7f9a6c0fb550 msgr2=0x7f9a6c069030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:31.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.321+0000 7f9a74364700 1 -- 172.21.15.105:0/1564704024 shutdown_connections 2024-04-06T02:47:31.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:31.321+0000 7f9a74364700 1 -- 172.21.15.105:0/1564704024 wait complete. 2024-04-06T02:47:31.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:31 smithi105 ceph-mon[29228]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:31.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:31 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/1564704024' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-06T02:47:31.974 INFO:tasks.cephadm:Setting up client nodes... 2024-04-06T02:47:31.975 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-06T02:47:31.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:31 smithi184 ceph-mon[37439]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:31.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:31 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/1564704024' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-06T02:47:32.179 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:47:33.573 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:33 smithi105 ceph-mon[29228]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:33.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:33 smithi184 ceph-mon[37439]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:33.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.836+0000 7f0946450700 1 -- 172.21.15.105:0/3611634875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0940100450 msgr2=0x7f09401008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:33.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.836+0000 7f0946450700 1 --2- 172.21.15.105:0/3611634875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0940100450 0x7f09401008b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0930009910 tx=0x7f0930009c20 comp rx=0 tx=0).stop 2024-04-06T02:47:33.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.836+0000 7f0946450700 1 -- 172.21.15.105:0/3611634875 shutdown_connections 2024-04-06T02:47:33.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.836+0000 7f0946450700 1 --2- 172.21.15.105:0/3611634875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0940100450 0x7f09401008b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:33.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.836+0000 7f0946450700 1 --2- 172.21.15.105:0/3611634875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09400ff250 0x7f09400ff670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:33.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.836+0000 7f0946450700 1 -- 172.21.15.105:0/3611634875 >> 172.21.15.105:0/3611634875 conn(0x7f09400fa7f0 msgr2=0x7f09400fcc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:33.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.836+0000 7f0946450700 1 -- 172.21.15.105:0/3611634875 shutdown_connections 2024-04-06T02:47:33.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.836+0000 7f0946450700 1 -- 172.21.15.105:0/3611634875 wait complete. 2024-04-06T02:47:33.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.837+0000 7f0946450700 1 Processor -- start 2024-04-06T02:47:33.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.837+0000 7f0946450700 1 -- start start 2024-04-06T02:47:33.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.837+0000 7f0946450700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f09400ff250 0x7f0940109270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:33.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.837+0000 7f0946450700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0940100450 0x7f094019b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:33.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.837+0000 7f0946450700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094019b670 con 0x7f09400ff250 2024-04-06T02:47:33.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.837+0000 7f0946450700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094019b7b0 con 0x7f0940100450 2024-04-06T02:47:33.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.838+0000 7f093ffff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f09400ff250 0x7f0940109270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:33.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.838+0000 7f093ffff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f09400ff250 0x7f0940109270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:58402/0 (socket says 172.21.15.105:58402) 2024-04-06T02:47:33.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.838+0000 7f093ffff700 1 -- 172.21.15.105:0/661273976 learned_addr learned my addr 172.21.15.105:0/661273976 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:47:33.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.838+0000 7f093f7fe700 1 --2- 172.21.15.105:0/661273976 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0940100450 0x7f094019b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:33.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.838+0000 7f093ffff700 1 -- 172.21.15.105:0/661273976 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0940100450 msgr2=0x7f094019b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:33.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.838+0000 7f093ffff700 1 --2- 172.21.15.105:0/661273976 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0940100450 0x7f094019b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:33.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.838+0000 7f093ffff700 1 -- 172.21.15.105:0/661273976 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09300095f0 con 0x7f09400ff250 2024-04-06T02:47:33.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.838+0000 7f093f7fe700 1 --2- 172.21.15.105:0/661273976 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0940100450 0x7f094019b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T02:47:33.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.838+0000 7f093ffff700 1 --2- 172.21.15.105:0/661273976 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f09400ff250 0x7f0940109270 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f093400e8f0 tx=0x7f093400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:33.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.838+0000 7f093d7fa700 1 -- 172.21.15.105:0/661273976 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0934009790 con 0x7f09400ff250 2024-04-06T02:47:33.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.838+0000 7f0946450700 1 -- 172.21.15.105:0/661273976 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f094019ba90 con 0x7f09400ff250 2024-04-06T02:47:33.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.838+0000 7f093d7fa700 1 -- 172.21.15.105:0/661273976 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f09340044d0 con 0x7f09400ff250 2024-04-06T02:47:33.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.838+0000 7f0946450700 1 -- 172.21.15.105:0/661273976 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f094019bfe0 con 0x7f09400ff250 2024-04-06T02:47:33.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.838+0000 7f093d7fa700 1 -- 172.21.15.105:0/661273976 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0934003dc0 con 0x7f09400ff250 2024-04-06T02:47:33.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.839+0000 7f0946450700 1 -- 172.21.15.105:0/661273976 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f092c005320 con 0x7f09400ff250 2024-04-06T02:47:33.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.840+0000 7f093d7fa700 1 -- 172.21.15.105:0/661273976 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f09340184b0 con 0x7f09400ff250 2024-04-06T02:47:33.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.840+0000 7f093d7fa700 1 --2- 172.21.15.105:0/661273976 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f092806c4e0 0x7f092806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:33.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.841+0000 7f093f7fe700 1 --2- 172.21.15.105:0/661273976 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f092806c4e0 0x7f092806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:33.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.841+0000 7f093d7fa700 1 -- 172.21.15.105:0/661273976 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f0934014070 con 0x7f09400ff250 2024-04-06T02:47:33.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.841+0000 7f093f7fe700 1 --2- 172.21.15.105:0/661273976 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f092806c4e0 0x7f092806e9a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f09300059d0 tx=0x7f0930005940 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:33.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:33.845+0000 7f093d7fa700 1 -- 172.21.15.105:0/661273976 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0934059e30 con 0x7f09400ff250 2024-04-06T02:47:34.094 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:34.107+0000 7f0946450700 1 -- 172.21.15.105:0/661273976 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f092c005190 con 0x7f09400ff250 2024-04-06T02:47:34.097 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:34.111+0000 7f093d7fa700 1 -- 172.21.15.105:0/661273976 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7f09340599c0 con 0x7f09400ff250 2024-04-06T02:47:34.098 INFO:teuthology.orchestra.run.smithi105.stdout:[client.0] 2024-04-06T02:47:34.098 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQDGtxBm/SWRBhAAGksommT3TUtvgVxzuFuR8g== 2024-04-06T02:47:34.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:34.113+0000 7f0946450700 1 -- 172.21.15.105:0/661273976 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f092806c4e0 msgr2=0x7f092806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:34.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:34.113+0000 7f0946450700 1 --2- 172.21.15.105:0/661273976 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f092806c4e0 0x7f092806e9a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f09300059d0 tx=0x7f0930005940 comp rx=0 tx=0).stop 2024-04-06T02:47:34.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:34.113+0000 7f0946450700 1 -- 172.21.15.105:0/661273976 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f09400ff250 msgr2=0x7f0940109270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:34.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:34.113+0000 7f0946450700 1 --2- 172.21.15.105:0/661273976 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f09400ff250 0x7f0940109270 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f093400e8f0 tx=0x7f093400ecb0 comp rx=0 tx=0).stop 2024-04-06T02:47:34.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:34.114+0000 7f0946450700 1 -- 172.21.15.105:0/661273976 shutdown_connections 2024-04-06T02:47:34.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:34.114+0000 7f0946450700 1 --2- 172.21.15.105:0/661273976 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f092806c4e0 0x7f092806e9a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:34.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:34.114+0000 7f0946450700 1 --2- 172.21.15.105:0/661273976 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f09400ff250 0x7f0940109270 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:34.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:34.114+0000 7f0946450700 1 --2- 172.21.15.105:0/661273976 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0940100450 0x7f094019b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:34.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:34.114+0000 7f0946450700 1 -- 172.21.15.105:0/661273976 >> 172.21.15.105:0/661273976 conn(0x7f09400fa7f0 msgr2=0x7f09400fc3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:34.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:34.114+0000 7f0946450700 1 -- 172.21.15.105:0/661273976 shutdown_connections 2024-04-06T02:47:34.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:34.114+0000 7f0946450700 1 -- 172.21.15.105:0/661273976 wait complete. 2024-04-06T02:47:34.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:34 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/661273976' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T02:47:34.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:34 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/661273976' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-06T02:47:34.719 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-06T02:47:34.720 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-06T02:47:34.720 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-06T02:47:34.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:34 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/661273976' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T02:47:34.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:34 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/661273976' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-06T02:47:34.761 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-06T02:47:34.988 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi184/config 2024-04-06T02:47:35.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:35 smithi105 ceph-mon[29228]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:35.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:35 smithi184 ceph-mon[37439]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:36.523 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.536+0000 7fa43cb0a700 1 -- 172.21.15.184:0/1900058135 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa438100b80 msgr2=0x7fa438100fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:36.523 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.536+0000 7fa43cb0a700 1 --2- 172.21.15.184:0/1900058135 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa438100b80 0x7fa438100fa0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa428009910 tx=0x7fa428009c20 comp rx=0 tx=0).stop 2024-04-06T02:47:36.523 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.536+0000 7fa43cb0a700 1 -- 172.21.15.184:0/1900058135 shutdown_connections 2024-04-06T02:47:36.523 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.536+0000 7fa43cb0a700 1 --2- 172.21.15.184:0/1900058135 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa438101d80 0x7fa4381021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:36.523 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.536+0000 7fa43cb0a700 1 --2- 172.21.15.184:0/1900058135 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa438100b80 0x7fa438100fa0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:36.523 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.536+0000 7fa43cb0a700 1 -- 172.21.15.184:0/1900058135 >> 172.21.15.184:0/1900058135 conn(0x7fa4380fc100 msgr2=0x7fa4380fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:36.523 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.536+0000 7fa43cb0a700 1 -- 172.21.15.184:0/1900058135 shutdown_connections 2024-04-06T02:47:36.523 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.536+0000 7fa43cb0a700 1 -- 172.21.15.184:0/1900058135 wait complete. 2024-04-06T02:47:36.523 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.537+0000 7fa43cb0a700 1 Processor -- start 2024-04-06T02:47:36.523 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.537+0000 7fa43cb0a700 1 -- start start 2024-04-06T02:47:36.524 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.537+0000 7fa43cb0a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa438100b80 0x7fa438196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:36.524 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.537+0000 7fa43cb0a700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa438101d80 0x7fa4381968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:36.524 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.537+0000 7fa43cb0a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa438196ee0 con 0x7fa438100b80 2024-04-06T02:47:36.524 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.537+0000 7fa43cb0a700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa438197020 con 0x7fa438101d80 2024-04-06T02:47:36.524 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.538+0000 7fa435d9b700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa438101d80 0x7fa4381968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:36.524 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.538+0000 7fa435d9b700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa438101d80 0x7fa4381968c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:58710/0 (socket says 172.21.15.184:58710) 2024-04-06T02:47:36.524 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.538+0000 7fa435d9b700 1 -- 172.21.15.184:0/2188614346 learned_addr learned my addr 172.21.15.184:0/2188614346 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-04-06T02:47:36.524 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.538+0000 7fa43659c700 1 --2- 172.21.15.184:0/2188614346 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa438100b80 0x7fa438196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:36.524 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.538+0000 7fa435d9b700 1 -- 172.21.15.184:0/2188614346 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa438100b80 msgr2=0x7fa438196380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:36.524 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.538+0000 7fa435d9b700 1 --2- 172.21.15.184:0/2188614346 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa438100b80 0x7fa438196380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:36.524 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.538+0000 7fa435d9b700 1 -- 172.21.15.184:0/2188614346 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4280095f0 con 0x7fa438101d80 2024-04-06T02:47:36.524 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.538+0000 7fa435d9b700 1 --2- 172.21.15.184:0/2188614346 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa438101d80 0x7fa4381968c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa42c00b4b0 tx=0x7fa42c00b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:36.524 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.538+0000 7fa4277fe700 1 -- 172.21.15.184:0/2188614346 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa42c011800 con 0x7fa438101d80 2024-04-06T02:47:36.524 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.538+0000 7fa43cb0a700 1 -- 172.21.15.184:0/2188614346 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa43819bad0 con 0x7fa438101d80 2024-04-06T02:47:36.525 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.538+0000 7fa4277fe700 1 -- 172.21.15.184:0/2188614346 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa42c011e10 con 0x7fa438101d80 2024-04-06T02:47:36.525 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.538+0000 7fa4277fe700 1 -- 172.21.15.184:0/2188614346 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa42c00f2e0 con 0x7fa438101d80 2024-04-06T02:47:36.525 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.538+0000 7fa43cb0a700 1 -- 172.21.15.184:0/2188614346 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa43819c0a0 con 0x7fa438101d80 2024-04-06T02:47:36.526 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.539+0000 7fa43cb0a700 1 -- 172.21.15.184:0/2188614346 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa438109760 con 0x7fa438101d80 2024-04-06T02:47:36.526 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.540+0000 7fa4277fe700 1 -- 172.21.15.184:0/2188614346 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fa42c010380 con 0x7fa438101d80 2024-04-06T02:47:36.526 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.540+0000 7fa4277fe700 1 --2- 172.21.15.184:0/2188614346 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fa42006c4e0 0x7fa42006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:36.526 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.540+0000 7fa4277fe700 1 -- 172.21.15.184:0/2188614346 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa42c08b130 con 0x7fa438101d80 2024-04-06T02:47:36.526 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.540+0000 7fa43659c700 1 --2- 172.21.15.184:0/2188614346 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fa42006c4e0 0x7fa42006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:36.526 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.541+0000 7fa43659c700 1 --2- 172.21.15.184:0/2188614346 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fa42006c4e0 0x7fa42006e9a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa428006010 tx=0x7fa428005920 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:36.529 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.545+0000 7fa4277fe700 1 -- 172.21.15.184:0/2188614346 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa42c0596e0 con 0x7fa438101d80 2024-04-06T02:47:36.792 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.807+0000 7fa43cb0a700 1 -- 172.21.15.184:0/2188614346 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fa43819c710 con 0x7fa438101d80 2024-04-06T02:47:36.799 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.814+0000 7fa4277fe700 1 -- 172.21.15.184:0/2188614346 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7fa42c059270 con 0x7fa438101d80 2024-04-06T02:47:36.799 INFO:teuthology.orchestra.run.smithi184.stdout:[client.1] 2024-04-06T02:47:36.799 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQDItxBmTcc6MBAAU0nN2/H2/OCS+ld33MZpPw== 2024-04-06T02:47:36.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.816+0000 7fa43cb0a700 1 -- 172.21.15.184:0/2188614346 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fa42006c4e0 msgr2=0x7fa42006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:36.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.816+0000 7fa43cb0a700 1 --2- 172.21.15.184:0/2188614346 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fa42006c4e0 0x7fa42006e9a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa428006010 tx=0x7fa428005920 comp rx=0 tx=0).stop 2024-04-06T02:47:36.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.816+0000 7fa43cb0a700 1 -- 172.21.15.184:0/2188614346 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa438101d80 msgr2=0x7fa4381968c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:36.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.816+0000 7fa43cb0a700 1 --2- 172.21.15.184:0/2188614346 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa438101d80 0x7fa4381968c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa42c00b4b0 tx=0x7fa42c00b870 comp rx=0 tx=0).stop 2024-04-06T02:47:36.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.816+0000 7fa43cb0a700 1 -- 172.21.15.184:0/2188614346 shutdown_connections 2024-04-06T02:47:36.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.816+0000 7fa43cb0a700 1 --2- 172.21.15.184:0/2188614346 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fa42006c4e0 0x7fa42006e9a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:36.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.816+0000 7fa43cb0a700 1 --2- 172.21.15.184:0/2188614346 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa438100b80 0x7fa438196380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:36.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.816+0000 7fa43cb0a700 1 --2- 172.21.15.184:0/2188614346 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa438101d80 0x7fa4381968c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:36.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.816+0000 7fa43cb0a700 1 -- 172.21.15.184:0/2188614346 >> 172.21.15.184:0/2188614346 conn(0x7fa4380fc100 msgr2=0x7fa438104fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:36.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.816+0000 7fa43cb0a700 1 -- 172.21.15.184:0/2188614346 shutdown_connections 2024-04-06T02:47:36.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:47:36.816+0000 7fa43cb0a700 1 -- 172.21.15.184:0/2188614346 wait complete. 2024-04-06T02:47:37.430 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-04-06T02:47:37.430 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-06T02:47:37.430 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-06T02:47:37.473 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-06T02:47:37.473 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-06T02:47:37.473 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph mgr dump --format=json 2024-04-06T02:47:37.690 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:47:37.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:37 smithi184 ceph-mon[37439]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:37.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:37 smithi184 ceph-mon[37439]: from='client.? 172.21.15.184:0/2188614346' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T02:47:37.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:37 smithi184 ceph-mon[37439]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T02:47:37.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:37 smithi184 ceph-mon[37439]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-06T02:47:37.808 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:37 smithi105 ceph-mon[29228]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:37.808 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:37 smithi105 ceph-mon[29228]: from='client.? 172.21.15.184:0/2188614346' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T02:47:37.808 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:37 smithi105 ceph-mon[29228]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T02:47:37.808 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:37 smithi105 ceph-mon[29228]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-06T02:47:39.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.415+0000 7f2e68acb700 1 -- 172.21.15.105:0/1451342949 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e64101d60 msgr2=0x7f2e64102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:39.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.415+0000 7f2e68acb700 1 --2- 172.21.15.105:0/1451342949 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e64101d60 0x7f2e64102140 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2e58009940 tx=0x7f2e58009c50 comp rx=0 tx=0).stop 2024-04-06T02:47:39.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.415+0000 7f2e68acb700 1 -- 172.21.15.105:0/1451342949 shutdown_connections 2024-04-06T02:47:39.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.415+0000 7f2e68acb700 1 --2- 172.21.15.105:0/1451342949 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e64102680 0x7f2e6410ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:39.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.415+0000 7f2e68acb700 1 --2- 172.21.15.105:0/1451342949 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e64101d60 0x7f2e64102140 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:39.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.415+0000 7f2e68acb700 1 -- 172.21.15.105:0/1451342949 >> 172.21.15.105:0/1451342949 conn(0x7f2e640fb570 msgr2=0x7f2e640fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:39.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.416+0000 7f2e68acb700 1 -- 172.21.15.105:0/1451342949 shutdown_connections 2024-04-06T02:47:39.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.416+0000 7f2e68acb700 1 -- 172.21.15.105:0/1451342949 wait complete. 2024-04-06T02:47:39.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.416+0000 7f2e68acb700 1 Processor -- start 2024-04-06T02:47:39.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.417+0000 7f2e68acb700 1 -- start start 2024-04-06T02:47:39.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.417+0000 7f2e68acb700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e64101d60 0x7f2e6419c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:39.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.417+0000 7f2e68acb700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e64102680 0x7f2e6419ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:39.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.417+0000 7f2e68acb700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6419d0e0 con 0x7f2e64102680 2024-04-06T02:47:39.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.417+0000 7f2e68acb700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e64196590 con 0x7f2e64101d60 2024-04-06T02:47:39.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.417+0000 7f2e61d9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e64102680 0x7f2e6419ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:39.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.417+0000 7f2e61d9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e64102680 0x7f2e6419ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:58426/0 (socket says 172.21.15.105:58426) 2024-04-06T02:47:39.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.417+0000 7f2e61d9b700 1 -- 172.21.15.105:0/800326112 learned_addr learned my addr 172.21.15.105:0/800326112 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:47:39.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.417+0000 7f2e6259c700 1 --2- 172.21.15.105:0/800326112 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e64101d60 0x7f2e6419c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:39.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.417+0000 7f2e61d9b700 1 -- 172.21.15.105:0/800326112 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e64101d60 msgr2=0x7f2e6419c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:39.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.417+0000 7f2e61d9b700 1 --2- 172.21.15.105:0/800326112 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e64101d60 0x7f2e6419c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:39.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.417+0000 7f2e61d9b700 1 -- 172.21.15.105:0/800326112 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e580095f0 con 0x7f2e64102680 2024-04-06T02:47:39.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.417+0000 7f2e61d9b700 1 --2- 172.21.15.105:0/800326112 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e64102680 0x7f2e6419ca50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2e5400c8f0 tx=0x7f2e5400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:39.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.417+0000 7f2e537fe700 1 -- 172.21.15.105:0/800326112 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e540041f0 con 0x7f2e64102680 2024-04-06T02:47:39.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.418+0000 7f2e68acb700 1 -- 172.21.15.105:0/800326112 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e64196870 con 0x7f2e64102680 2024-04-06T02:47:39.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.418+0000 7f2e68acb700 1 -- 172.21.15.105:0/800326112 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e64196dc0 con 0x7f2e64102680 2024-04-06T02:47:39.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.418+0000 7f2e537fe700 1 -- 172.21.15.105:0/800326112 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2e54004350 con 0x7f2e64102680 2024-04-06T02:47:39.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.418+0000 7f2e537fe700 1 -- 172.21.15.105:0/800326112 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e54003c30 con 0x7f2e64102680 2024-04-06T02:47:39.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.419+0000 7f2e68acb700 1 -- 172.21.15.105:0/800326112 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e44005320 con 0x7f2e64102680 2024-04-06T02:47:39.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.419+0000 7f2e537fe700 1 -- 172.21.15.105:0/800326112 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f2e5400f450 con 0x7f2e64102680 2024-04-06T02:47:39.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.420+0000 7f2e537fe700 1 --2- 172.21.15.105:0/800326112 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f2e4c06c4e0 0x7f2e4c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:39.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.420+0000 7f2e537fe700 1 -- 172.21.15.105:0/800326112 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f2e5408aaf0 con 0x7f2e64102680 2024-04-06T02:47:39.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.420+0000 7f2e6259c700 1 --2- 172.21.15.105:0/800326112 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f2e4c06c4e0 0x7f2e4c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:39.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.420+0000 7f2e6259c700 1 --2- 172.21.15.105:0/800326112 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f2e4c06c4e0 0x7f2e4c06e9a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2e58005f50 tx=0x7f2e58005dc0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:39.409 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.424+0000 7f2e537fe700 1 -- 172.21.15.105:0/800326112 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2e54059120 con 0x7f2e64102680 2024-04-06T02:47:39.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.668+0000 7f2e68acb700 1 -- 172.21.15.105:0/800326112 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f2e44005190 con 0x7f2e64102680 2024-04-06T02:47:39.657 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.672+0000 7f2e537fe700 1 -- 172.21.15.105:0/800326112 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+173048 (secure 0 0 0) 0x7f2e54058cb0 con 0x7f2e64102680 2024-04-06T02:47:39.660 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:47:39.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.675+0000 7f2e68acb700 1 -- 172.21.15.105:0/800326112 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f2e4c06c4e0 msgr2=0x7f2e4c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:39.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.675+0000 7f2e68acb700 1 --2- 172.21.15.105:0/800326112 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f2e4c06c4e0 0x7f2e4c06e9a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2e58005f50 tx=0x7f2e58005dc0 comp rx=0 tx=0).stop 2024-04-06T02:47:39.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.675+0000 7f2e68acb700 1 -- 172.21.15.105:0/800326112 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e64102680 msgr2=0x7f2e6419ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:39.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.675+0000 7f2e68acb700 1 --2- 172.21.15.105:0/800326112 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e64102680 0x7f2e6419ca50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2e5400c8f0 tx=0x7f2e5400cc00 comp rx=0 tx=0).stop 2024-04-06T02:47:39.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.675+0000 7f2e68acb700 1 -- 172.21.15.105:0/800326112 shutdown_connections 2024-04-06T02:47:39.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.675+0000 7f2e68acb700 1 --2- 172.21.15.105:0/800326112 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f2e4c06c4e0 0x7f2e4c06e9a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:39.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.675+0000 7f2e68acb700 1 --2- 172.21.15.105:0/800326112 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e64101d60 0x7f2e6419c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:39.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.675+0000 7f2e68acb700 1 --2- 172.21.15.105:0/800326112 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e64102680 0x7f2e6419ca50 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:39.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.675+0000 7f2e68acb700 1 -- 172.21.15.105:0/800326112 >> 172.21.15.105:0/800326112 conn(0x7f2e640fb570 msgr2=0x7f2e641053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:39.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.675+0000 7f2e68acb700 1 -- 172.21.15.105:0/800326112 shutdown_connections 2024-04-06T02:47:39.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:39.675+0000 7f2e68acb700 1 -- 172.21.15.105:0/800326112 wait complete. 2024-04-06T02:47:39.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:39 smithi105 ceph-mon[29228]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:39.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:39 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:47:39.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:39 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/800326112' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-06T02:47:39.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:39 smithi184 ceph-mon[37439]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:39.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:39 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:47:39.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:39 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/800326112' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-06T02:47:40.351 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":19,"active_gid":14221,"active_name":"smithi105.vqqcpx","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2334550011},{"type":"v1","addr":"172.21.15.105:6801","nonce":2334550011}]},"active_addr":"172.21.15.105:6801/2334550011","active_change":"2024-04-06T02:43:54.575139+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":14244,"name":"smithi184.fzhhql","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.105:8443/","prometheus":"http://172.21.15.105:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.105:0","nonce":2302199122}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.105:0","nonce":1031635854}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.105:0","nonce":3801418786}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.105:0","nonce":1860420844}]}]} 2024-04-06T02:47:40.355 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-06T02:47:40.355 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-06T02:47:40.355 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-06T02:47:40.560 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:47:41.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:41 smithi105 ceph-mon[29228]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:41.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:41 smithi184 ceph-mon[37439]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:42.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.205+0000 7f710c234700 1 -- 172.21.15.105:0/288398008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7104101d80 msgr2=0x7f7104102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:42.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.205+0000 7f710c234700 1 --2- 172.21.15.105:0/288398008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7104101d80 0x7f7104102160 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7100009910 tx=0x7f7100009c20 comp rx=0 tx=0).stop 2024-04-06T02:47:42.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.205+0000 7f710c234700 1 -- 172.21.15.105:0/288398008 shutdown_connections 2024-04-06T02:47:42.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.205+0000 7f710c234700 1 --2- 172.21.15.105:0/288398008 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f71041026a0 0x7f710410ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:42.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.205+0000 7f710c234700 1 --2- 172.21.15.105:0/288398008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7104101d80 0x7f7104102160 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:42.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.205+0000 7f710c234700 1 -- 172.21.15.105:0/288398008 >> 172.21.15.105:0/288398008 conn(0x7f71040fb550 msgr2=0x7f71040fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:42.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.205+0000 7f710c234700 1 -- 172.21.15.105:0/288398008 shutdown_connections 2024-04-06T02:47:42.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.205+0000 7f710c234700 1 -- 172.21.15.105:0/288398008 wait complete. 2024-04-06T02:47:42.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.205+0000 7f710c234700 1 Processor -- start 2024-04-06T02:47:42.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.205+0000 7f710c234700 1 -- start start 2024-04-06T02:47:42.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.206+0000 7f710c234700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7104101d80 0x7f710419c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:42.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.206+0000 7f710c234700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71041026a0 0x7f710419ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:42.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.206+0000 7f710c234700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710419d0c0 con 0x7f71041026a0 2024-04-06T02:47:42.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.206+0000 7f710c234700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7104196570 con 0x7f7104101d80 2024-04-06T02:47:42.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.206+0000 7f71097cf700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71041026a0 0x7f710419ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:42.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.206+0000 7f71097cf700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71041026a0 0x7f710419ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:32912/0 (socket says 172.21.15.105:32912) 2024-04-06T02:47:42.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.206+0000 7f71097cf700 1 -- 172.21.15.105:0/2146711522 learned_addr learned my addr 172.21.15.105:0/2146711522 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:47:42.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.206+0000 7f7109fd0700 1 --2- 172.21.15.105:0/2146711522 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7104101d80 0x7f710419c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:42.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.206+0000 7f71097cf700 1 -- 172.21.15.105:0/2146711522 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7104101d80 msgr2=0x7f710419c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:42.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.206+0000 7f71097cf700 1 --2- 172.21.15.105:0/2146711522 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7104101d80 0x7f710419c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:42.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.206+0000 7f71097cf700 1 -- 172.21.15.105:0/2146711522 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71000095f0 con 0x7f71041026a0 2024-04-06T02:47:42.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.206+0000 7f71097cf700 1 --2- 172.21.15.105:0/2146711522 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71041026a0 0x7f710419ca30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f70f400d900 tx=0x7f70f400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:42.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.206+0000 7f70faffd700 1 -- 172.21.15.105:0/2146711522 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70f4009900 con 0x7f71041026a0 2024-04-06T02:47:42.194 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.206+0000 7f70faffd700 1 -- 172.21.15.105:0/2146711522 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f70f4010460 con 0x7f71041026a0 2024-04-06T02:47:42.194 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.206+0000 7f710c234700 1 -- 172.21.15.105:0/2146711522 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7104196850 con 0x7f71041026a0 2024-04-06T02:47:42.194 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.206+0000 7f70faffd700 1 -- 172.21.15.105:0/2146711522 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70f400f5f0 con 0x7f71041026a0 2024-04-06T02:47:42.194 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.206+0000 7f710c234700 1 -- 172.21.15.105:0/2146711522 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7104196da0 con 0x7f71041026a0 2024-04-06T02:47:42.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.207+0000 7f710c234700 1 -- 172.21.15.105:0/2146711522 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7104108290 con 0x7f71041026a0 2024-04-06T02:47:42.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.208+0000 7f70faffd700 1 -- 172.21.15.105:0/2146711522 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f70f40105d0 con 0x7f71041026a0 2024-04-06T02:47:42.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.208+0000 7f70faffd700 1 --2- 172.21.15.105:0/2146711522 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f70f006c4e0 0x7f70f006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:42.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.208+0000 7f70faffd700 1 -- 172.21.15.105:0/2146711522 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f70f408b080 con 0x7f71041026a0 2024-04-06T02:47:42.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.208+0000 7f7109fd0700 1 --2- 172.21.15.105:0/2146711522 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f70f006c4e0 0x7f70f006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:42.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.209+0000 7f7109fd0700 1 --2- 172.21.15.105:0/2146711522 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f70f006c4e0 0x7f70f006e9a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7100006010 tx=0x7f710001a040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:42.198 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.213+0000 7f70faffd700 1 -- 172.21.15.105:0/2146711522 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f70f4059630 con 0x7f71041026a0 2024-04-06T02:47:42.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.402+0000 7f710c234700 1 -- 172.21.15.105:0/2146711522 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7104197040 con 0x7f71041026a0 2024-04-06T02:47:42.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.403+0000 7f70faffd700 1 -- 172.21.15.105:0/2146711522 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11389 (secure 0 0 0) 0x7f70f40591c0 con 0x7f71041026a0 2024-04-06T02:47:42.388 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:47:42.388 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":41,"fsid":"130df578-f3bf-11ee-b649-cb9ed24678a4","created":"2024-04-06T02:41:32.644992+0000","modified":"2024-04-06T02:47:26.582254+0000","last_up_change":"2024-04-06T02:47:25.576978+0000","last_in_change":"2024-04-06T02:47:06.634446+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T02:46:12.900794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41c20c92-5314-4354-b659-cb449ef98c81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2018109658},{"type":"v1","addr":"172.21.15.105:6803","nonce":2018109658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2018109658},{"type":"v1","addr":"172.21.15.105:6805","nonce":2018109658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":2018109658},{"type":"v1","addr":"172.21.15.105:6809","nonce":2018109658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2018109658},{"type":"v1","addr":"172.21.15.105:6807","nonce":2018109658}]},"public_addr":"172.21.15.105:6803/2018109658","cluster_addr":"172.21.15.105:6805/2018109658","heartbeat_back_addr":"172.21.15.105:6809/2018109658","heartbeat_front_addr":"172.21.15.105:6807/2018109658","state":["exists","up"]},{"osd":1,"uuid":"ea3f1b19-62f8-488e-b62a-2340aef1112a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":1345191629},{"type":"v1","addr":"172.21.15.105:6811","nonce":1345191629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":1345191629},{"type":"v1","addr":"172.21.15.105:6813","nonce":1345191629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":1345191629},{"type":"v1","addr":"172.21.15.105:6817","nonce":1345191629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":1345191629},{"type":"v1","addr":"172.21.15.105:6815","nonce":1345191629}]},"public_addr":"172.21.15.105:6811/1345191629","cluster_addr":"172.21.15.105:6813/1345191629","heartbeat_back_addr":"172.21.15.105:6817/1345191629","heartbeat_front_addr":"172.21.15.105:6815/1345191629","state":["exists","up"]},{"osd":2,"uuid":"def7693a-bb30-4c74-ae20-59a7a86ef9db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":3651690266},{"type":"v1","addr":"172.21.15.105:6819","nonce":3651690266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":3651690266},{"type":"v1","addr":"172.21.15.105:6821","nonce":3651690266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3651690266},{"type":"v1","addr":"172.21.15.105:6825","nonce":3651690266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":3651690266},{"type":"v1","addr":"172.21.15.105:6823","nonce":3651690266}]},"public_addr":"172.21.15.105:6819/3651690266","cluster_addr":"172.21.15.105:6821/3651690266","heartbeat_back_addr":"172.21.15.105:6825/3651690266","heartbeat_front_addr":"172.21.15.105:6823/3651690266","state":["exists","up"]},{"osd":3,"uuid":"97f959d7-5922-4423-9d63-e2fe677a7019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":330017535},{"type":"v1","addr":"172.21.15.184:6801","nonce":330017535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":330017535},{"type":"v1","addr":"172.21.15.184:6803","nonce":330017535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":330017535},{"type":"v1","addr":"172.21.15.184:6807","nonce":330017535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":330017535},{"type":"v1","addr":"172.21.15.184:6805","nonce":330017535}]},"public_addr":"172.21.15.184:6801/330017535","cluster_addr":"172.21.15.184:6803/330017535","heartbeat_back_addr":"172.21.15.184:6807/330017535","heartbeat_front_addr":"172.21.15.184:6805/330017535","state":["exists","up"]},{"osd":4,"uuid":"34d7ebbd-c725-4948-ae33-3ffb40ec9979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2237051966},{"type":"v1","addr":"172.21.15.184:6809","nonce":2237051966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2237051966},{"type":"v1","addr":"172.21.15.184:6811","nonce":2237051966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2237051966},{"type":"v1","addr":"172.21.15.184:6815","nonce":2237051966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2237051966},{"type":"v1","addr":"172.21.15.184:6813","nonce":2237051966}]},"public_addr":"172.21.15.184:6809/2237051966","cluster_addr":"172.21.15.184:6811/2237051966","heartbeat_back_addr":"172.21.15.184:6815/2237051966","heartbeat_front_addr":"172.21.15.184:6813/2237051966","state":["exists","up"]},{"osd":5,"uuid":"95e64c1c-1f14-4017-b27c-11f447ac5a67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":1371998009},{"type":"v1","addr":"172.21.15.184:6817","nonce":1371998009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":1371998009},{"type":"v1","addr":"172.21.15.184:6819","nonce":1371998009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":1371998009},{"type":"v1","addr":"172.21.15.184:6823","nonce":1371998009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":1371998009},{"type":"v1","addr":"172.21.15.184:6821","nonce":1371998009}]},"public_addr":"172.21.15.184:6817/1371998009","cluster_addr":"172.21.15.184:6819/1371998009","heartbeat_back_addr":"172.21.15.184:6823/1371998009","heartbeat_front_addr":"172.21.15.184:6821/1371998009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:45:12.515625+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:45:41.117291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:46:09.180416+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:46:33.131470+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:46:58.464606+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:47:24.027273+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:0/2231570326":"2024-04-07T02:43:54.575029+0000","172.21.15.105:0/1342777414":"2024-04-07T02:43:54.575029+0000","172.21.15.105:6800/1949998385":"2024-04-07T02:43:54.575029+0000","172.21.15.105:6801/1394104199":"2024-04-07T02:42:35.032003+0000","172.21.15.105:6800/1394104199":"2024-04-07T02:42:35.032003+0000","172.21.15.105:0/2131480441":"2024-04-07T02:42:35.032003+0000","172.21.15.105:0/1539671918":"2024-04-07T02:42:02.154709+0000","172.21.15.105:0/1826925654":"2024-04-07T02:42:02.154709+0000","172.21.15.105:6801/3245193613":"2024-04-07T02:42:02.154709+0000","172.21.15.105:0/403675292":"2024-04-07T02:43:54.575029+0000","172.21.15.105:6800/3245193613":"2024-04-07T02:42:02.154709+0000","172.21.15.105:0/1323549385":"2024-04-07T02:42:02.154709+0000","172.21.15.105:6801/1949998385":"2024-04-07T02:43:54.575029+0000","172.21.15.105:0/1155423922":"2024-04-07T02:42:35.032003+0000","172.21.15.105:0/3833744400":"2024-04-07T02:42:35.032003+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T02:47:42.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.405+0000 7f710c234700 1 -- 172.21.15.105:0/2146711522 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f70f006c4e0 msgr2=0x7f70f006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:42.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.405+0000 7f710c234700 1 --2- 172.21.15.105:0/2146711522 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f70f006c4e0 0x7f70f006e9a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7100006010 tx=0x7f710001a040 comp rx=0 tx=0).stop 2024-04-06T02:47:42.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.405+0000 7f710c234700 1 -- 172.21.15.105:0/2146711522 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71041026a0 msgr2=0x7f710419ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:42.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.405+0000 7f710c234700 1 --2- 172.21.15.105:0/2146711522 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71041026a0 0x7f710419ca30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f70f400d900 tx=0x7f70f400dc10 comp rx=0 tx=0).stop 2024-04-06T02:47:42.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.405+0000 7f710c234700 1 -- 172.21.15.105:0/2146711522 shutdown_connections 2024-04-06T02:47:42.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.405+0000 7f710c234700 1 --2- 172.21.15.105:0/2146711522 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f70f006c4e0 0x7f70f006e9a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:42.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.405+0000 7f710c234700 1 --2- 172.21.15.105:0/2146711522 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7104101d80 0x7f710419c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:42.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.405+0000 7f710c234700 1 --2- 172.21.15.105:0/2146711522 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71041026a0 0x7f710419ca30 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:42.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.405+0000 7f710c234700 1 -- 172.21.15.105:0/2146711522 >> 172.21.15.105:0/2146711522 conn(0x7f71040fb550 msgr2=0x7f71041053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:42.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.405+0000 7f710c234700 1 -- 172.21.15.105:0/2146711522 shutdown_connections 2024-04-06T02:47:42.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:42.405+0000 7f710c234700 1 -- 172.21.15.105:0/2146711522 wait complete. 2024-04-06T02:47:42.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:42 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/2146711522' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T02:47:42.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:42 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/2146711522' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T02:47:43.151 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-06T02:47:43.151 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-06T02:47:43.355 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:47:43.732 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:43 smithi105 ceph-mon[29228]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:43.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:43 smithi184 ceph-mon[37439]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:45.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.063+0000 7fd7480bb700 1 -- 172.21.15.105:0/3682929592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd740100dc0 msgr2=0x7fd7401011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:45.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.063+0000 7fd7480bb700 1 --2- 172.21.15.105:0/3682929592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd740100dc0 0x7fd7401011a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd738009910 tx=0x7fd738009c20 comp rx=0 tx=0).stop 2024-04-06T02:47:45.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.063+0000 7fd7480bb700 1 -- 172.21.15.105:0/3682929592 shutdown_connections 2024-04-06T02:47:45.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.063+0000 7fd7480bb700 1 --2- 172.21.15.105:0/3682929592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd740101770 0x7fd7401057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:45.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.063+0000 7fd7480bb700 1 --2- 172.21.15.105:0/3682929592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd740100dc0 0x7fd7401011a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:45.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.063+0000 7fd7480bb700 1 -- 172.21.15.105:0/3682929592 >> 172.21.15.105:0/3682929592 conn(0x7fd7400fc650 msgr2=0x7fd7400fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:45.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.063+0000 7fd7480bb700 1 -- 172.21.15.105:0/3682929592 shutdown_connections 2024-04-06T02:47:45.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.063+0000 7fd7480bb700 1 -- 172.21.15.105:0/3682929592 wait complete. 2024-04-06T02:47:45.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.064+0000 7fd7480bb700 1 Processor -- start 2024-04-06T02:47:45.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.064+0000 7fd7480bb700 1 -- start start 2024-04-06T02:47:45.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.064+0000 7fd7480bb700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd740100dc0 0x7fd74019c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:45.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.064+0000 7fd7480bb700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd740101770 0x7fd74019cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:45.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.064+0000 7fd7480bb700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74019d1e0 con 0x7fd740100dc0 2024-04-06T02:47:45.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.064+0000 7fd7480bb700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7401966e0 con 0x7fd740101770 2024-04-06T02:47:45.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.064+0000 7fd745e57700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd740100dc0 0x7fd74019c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:45.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.064+0000 7fd745e57700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd740100dc0 0x7fd74019c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:32932/0 (socket says 172.21.15.105:32932) 2024-04-06T02:47:45.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.064+0000 7fd745e57700 1 -- 172.21.15.105:0/2018505124 learned_addr learned my addr 172.21.15.105:0/2018505124 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:47:45.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.065+0000 7fd745656700 1 --2- 172.21.15.105:0/2018505124 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd740101770 0x7fd74019cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:45.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.065+0000 7fd745e57700 1 -- 172.21.15.105:0/2018505124 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd740101770 msgr2=0x7fd74019cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:45.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.065+0000 7fd745e57700 1 --2- 172.21.15.105:0/2018505124 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd740101770 0x7fd74019cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:45.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.065+0000 7fd745e57700 1 -- 172.21.15.105:0/2018505124 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7380095f0 con 0x7fd740100dc0 2024-04-06T02:47:45.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.065+0000 7fd745e57700 1 --2- 172.21.15.105:0/2018505124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd740100dc0 0x7fd74019c610 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd738006010 tx=0x7fd73800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:45.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.065+0000 7fd736ffd700 1 -- 172.21.15.105:0/2018505124 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd73801d070 con 0x7fd740100dc0 2024-04-06T02:47:45.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.065+0000 7fd736ffd700 1 -- 172.21.15.105:0/2018505124 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd73800f460 con 0x7fd740100dc0 2024-04-06T02:47:45.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.065+0000 7fd7480bb700 1 -- 172.21.15.105:0/2018505124 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd740196960 con 0x7fd740100dc0 2024-04-06T02:47:45.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.065+0000 7fd736ffd700 1 -- 172.21.15.105:0/2018505124 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd738021640 con 0x7fd740100dc0 2024-04-06T02:47:45.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.065+0000 7fd7480bb700 1 -- 172.21.15.105:0/2018505124 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd740196e50 con 0x7fd740100dc0 2024-04-06T02:47:45.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.066+0000 7fd7480bb700 1 -- 172.21.15.105:0/2018505124 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd740109160 con 0x7fd740100dc0 2024-04-06T02:47:45.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.066+0000 7fd736ffd700 1 -- 172.21.15.105:0/2018505124 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fd73800faf0 con 0x7fd740100dc0 2024-04-06T02:47:45.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.067+0000 7fd736ffd700 1 --2- 172.21.15.105:0/2018505124 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fd72c06c4e0 0x7fd72c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:45.054 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.067+0000 7fd736ffd700 1 -- 172.21.15.105:0/2018505124 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fd73808c870 con 0x7fd740100dc0 2024-04-06T02:47:45.054 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.067+0000 7fd745656700 1 --2- 172.21.15.105:0/2018505124 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fd72c06c4e0 0x7fd72c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:45.054 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.067+0000 7fd745656700 1 --2- 172.21.15.105:0/2018505124 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fd72c06c4e0 0x7fd72c06e9a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd730006fd0 tx=0x7fd730008040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:45.056 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.071+0000 7fd736ffd700 1 -- 172.21.15.105:0/2018505124 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd73805ae20 con 0x7fd740100dc0 2024-04-06T02:47:45.247 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.262+0000 7fd7480bb700 1 -- 172.21.15.105:0/2018505124 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd74002ccc0 con 0x7fd740100dc0 2024-04-06T02:47:45.248 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.263+0000 7fd736ffd700 1 -- 172.21.15.105:0/2018505124 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11389 (secure 0 0 0) 0x7fd73805a9b0 con 0x7fd740100dc0 2024-04-06T02:47:45.249 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:47:45.249 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":41,"fsid":"130df578-f3bf-11ee-b649-cb9ed24678a4","created":"2024-04-06T02:41:32.644992+0000","modified":"2024-04-06T02:47:26.582254+0000","last_up_change":"2024-04-06T02:47:25.576978+0000","last_in_change":"2024-04-06T02:47:06.634446+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T02:46:12.900794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41c20c92-5314-4354-b659-cb449ef98c81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2018109658},{"type":"v1","addr":"172.21.15.105:6803","nonce":2018109658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2018109658},{"type":"v1","addr":"172.21.15.105:6805","nonce":2018109658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":2018109658},{"type":"v1","addr":"172.21.15.105:6809","nonce":2018109658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2018109658},{"type":"v1","addr":"172.21.15.105:6807","nonce":2018109658}]},"public_addr":"172.21.15.105:6803/2018109658","cluster_addr":"172.21.15.105:6805/2018109658","heartbeat_back_addr":"172.21.15.105:6809/2018109658","heartbeat_front_addr":"172.21.15.105:6807/2018109658","state":["exists","up"]},{"osd":1,"uuid":"ea3f1b19-62f8-488e-b62a-2340aef1112a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":1345191629},{"type":"v1","addr":"172.21.15.105:6811","nonce":1345191629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":1345191629},{"type":"v1","addr":"172.21.15.105:6813","nonce":1345191629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":1345191629},{"type":"v1","addr":"172.21.15.105:6817","nonce":1345191629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":1345191629},{"type":"v1","addr":"172.21.15.105:6815","nonce":1345191629}]},"public_addr":"172.21.15.105:6811/1345191629","cluster_addr":"172.21.15.105:6813/1345191629","heartbeat_back_addr":"172.21.15.105:6817/1345191629","heartbeat_front_addr":"172.21.15.105:6815/1345191629","state":["exists","up"]},{"osd":2,"uuid":"def7693a-bb30-4c74-ae20-59a7a86ef9db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":3651690266},{"type":"v1","addr":"172.21.15.105:6819","nonce":3651690266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":3651690266},{"type":"v1","addr":"172.21.15.105:6821","nonce":3651690266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3651690266},{"type":"v1","addr":"172.21.15.105:6825","nonce":3651690266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":3651690266},{"type":"v1","addr":"172.21.15.105:6823","nonce":3651690266}]},"public_addr":"172.21.15.105:6819/3651690266","cluster_addr":"172.21.15.105:6821/3651690266","heartbeat_back_addr":"172.21.15.105:6825/3651690266","heartbeat_front_addr":"172.21.15.105:6823/3651690266","state":["exists","up"]},{"osd":3,"uuid":"97f959d7-5922-4423-9d63-e2fe677a7019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":330017535},{"type":"v1","addr":"172.21.15.184:6801","nonce":330017535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":330017535},{"type":"v1","addr":"172.21.15.184:6803","nonce":330017535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":330017535},{"type":"v1","addr":"172.21.15.184:6807","nonce":330017535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":330017535},{"type":"v1","addr":"172.21.15.184:6805","nonce":330017535}]},"public_addr":"172.21.15.184:6801/330017535","cluster_addr":"172.21.15.184:6803/330017535","heartbeat_back_addr":"172.21.15.184:6807/330017535","heartbeat_front_addr":"172.21.15.184:6805/330017535","state":["exists","up"]},{"osd":4,"uuid":"34d7ebbd-c725-4948-ae33-3ffb40ec9979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2237051966},{"type":"v1","addr":"172.21.15.184:6809","nonce":2237051966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2237051966},{"type":"v1","addr":"172.21.15.184:6811","nonce":2237051966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2237051966},{"type":"v1","addr":"172.21.15.184:6815","nonce":2237051966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2237051966},{"type":"v1","addr":"172.21.15.184:6813","nonce":2237051966}]},"public_addr":"172.21.15.184:6809/2237051966","cluster_addr":"172.21.15.184:6811/2237051966","heartbeat_back_addr":"172.21.15.184:6815/2237051966","heartbeat_front_addr":"172.21.15.184:6813/2237051966","state":["exists","up"]},{"osd":5,"uuid":"95e64c1c-1f14-4017-b27c-11f447ac5a67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":1371998009},{"type":"v1","addr":"172.21.15.184:6817","nonce":1371998009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":1371998009},{"type":"v1","addr":"172.21.15.184:6819","nonce":1371998009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":1371998009},{"type":"v1","addr":"172.21.15.184:6823","nonce":1371998009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":1371998009},{"type":"v1","addr":"172.21.15.184:6821","nonce":1371998009}]},"public_addr":"172.21.15.184:6817/1371998009","cluster_addr":"172.21.15.184:6819/1371998009","heartbeat_back_addr":"172.21.15.184:6823/1371998009","heartbeat_front_addr":"172.21.15.184:6821/1371998009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:45:12.515625+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:45:41.117291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:46:09.180416+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:46:33.131470+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:46:58.464606+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:47:24.027273+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:0/2231570326":"2024-04-07T02:43:54.575029+0000","172.21.15.105:0/1342777414":"2024-04-07T02:43:54.575029+0000","172.21.15.105:6800/1949998385":"2024-04-07T02:43:54.575029+0000","172.21.15.105:6801/1394104199":"2024-04-07T02:42:35.032003+0000","172.21.15.105:6800/1394104199":"2024-04-07T02:42:35.032003+0000","172.21.15.105:0/2131480441":"2024-04-07T02:42:35.032003+0000","172.21.15.105:0/1539671918":"2024-04-07T02:42:02.154709+0000","172.21.15.105:0/1826925654":"2024-04-07T02:42:02.154709+0000","172.21.15.105:6801/3245193613":"2024-04-07T02:42:02.154709+0000","172.21.15.105:0/403675292":"2024-04-07T02:43:54.575029+0000","172.21.15.105:6800/3245193613":"2024-04-07T02:42:02.154709+0000","172.21.15.105:0/1323549385":"2024-04-07T02:42:02.154709+0000","172.21.15.105:6801/1949998385":"2024-04-07T02:43:54.575029+0000","172.21.15.105:0/1155423922":"2024-04-07T02:42:35.032003+0000","172.21.15.105:0/3833744400":"2024-04-07T02:42:35.032003+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T02:47:45.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.265+0000 7fd7480bb700 1 -- 172.21.15.105:0/2018505124 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fd72c06c4e0 msgr2=0x7fd72c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:45.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.265+0000 7fd7480bb700 1 --2- 172.21.15.105:0/2018505124 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fd72c06c4e0 0x7fd72c06e9a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd730006fd0 tx=0x7fd730008040 comp rx=0 tx=0).stop 2024-04-06T02:47:45.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.265+0000 7fd7480bb700 1 -- 172.21.15.105:0/2018505124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd740100dc0 msgr2=0x7fd74019c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:45.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.265+0000 7fd7480bb700 1 --2- 172.21.15.105:0/2018505124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd740100dc0 0x7fd74019c610 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd738006010 tx=0x7fd73800be20 comp rx=0 tx=0).stop 2024-04-06T02:47:45.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.265+0000 7fd7480bb700 1 -- 172.21.15.105:0/2018505124 shutdown_connections 2024-04-06T02:47:45.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.265+0000 7fd7480bb700 1 --2- 172.21.15.105:0/2018505124 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fd72c06c4e0 0x7fd72c06e9a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:45.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.265+0000 7fd7480bb700 1 --2- 172.21.15.105:0/2018505124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd740100dc0 0x7fd74019c610 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:45.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.265+0000 7fd7480bb700 1 --2- 172.21.15.105:0/2018505124 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd740101770 0x7fd74019cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:45.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.265+0000 7fd7480bb700 1 -- 172.21.15.105:0/2018505124 >> 172.21.15.105:0/2018505124 conn(0x7fd7400fc650 msgr2=0x7fd7400fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:45.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.266+0000 7fd7480bb700 1 -- 172.21.15.105:0/2018505124 shutdown_connections 2024-04-06T02:47:45.252 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:45.266+0000 7fd7480bb700 1 -- 172.21.15.105:0/2018505124 wait complete. 2024-04-06T02:47:45.783 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:45 smithi184 ceph-mon[37439]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:45.784 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:45 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/2018505124' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T02:47:45.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:45 smithi105 ceph-mon[29228]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:45.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:45 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/2018505124' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T02:47:46.017 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph tell osd.0 flush_pg_stats 2024-04-06T02:47:46.059 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph tell osd.1 flush_pg_stats 2024-04-06T02:47:46.059 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph tell osd.2 flush_pg_stats 2024-04-06T02:47:46.060 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph tell osd.3 flush_pg_stats 2024-04-06T02:47:46.060 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph tell osd.4 flush_pg_stats 2024-04-06T02:47:46.061 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph tell osd.5 flush_pg_stats 2024-04-06T02:47:46.323 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:47:46.361 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:47:46.954 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:47:47.821 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:47 smithi105 ceph-mon[29228]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:47.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:47 smithi184 ceph-mon[37439]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:48.094 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:47:48.095 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:47:48.095 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:47:49.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.203+0000 7f33f1f4f700 1 -- 172.21.15.105:0/2230661049 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33ec0ffa40 msgr2=0x7f33ec0ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:49.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.203+0000 7f33f1f4f700 1 --2- 172.21.15.105:0/2230661049 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33ec0ffa40 0x7f33ec0ffec0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f33d4009910 tx=0x7f33d4009c20 comp rx=0 tx=0).stop 2024-04-06T02:47:49.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.203+0000 7f33f1f4f700 1 -- 172.21.15.105:0/2230661049 shutdown_connections 2024-04-06T02:47:49.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.203+0000 7f33f1f4f700 1 --2- 172.21.15.105:0/2230661049 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33ec0ffa40 0x7f33ec0ffec0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:49.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.203+0000 7f33f1f4f700 1 --2- 172.21.15.105:0/2230661049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33ec105a40 0x7f33ec105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:49.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.203+0000 7f33f1f4f700 1 -- 172.21.15.105:0/2230661049 >> 172.21.15.105:0/2230661049 conn(0x7f33ec0fb570 msgr2=0x7f33ec0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:49.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.203+0000 7f33f1f4f700 1 -- 172.21.15.105:0/2230661049 shutdown_connections 2024-04-06T02:47:49.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.203+0000 7f33f1f4f700 1 -- 172.21.15.105:0/2230661049 wait complete. 2024-04-06T02:47:49.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.204+0000 7f33f1f4f700 1 Processor -- start 2024-04-06T02:47:49.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.204+0000 7f33f1f4f700 1 -- start start 2024-04-06T02:47:49.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.204+0000 7f33f1f4f700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33ec0ffa40 0x7f33ec1069b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:49.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.204+0000 7f33f1f4f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33ec105a40 0x7f33ec106ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:49.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.204+0000 7f33f1f4f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec107430 con 0x7f33ec105a40 2024-04-06T02:47:49.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.204+0000 7f33f1f4f700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec107570 con 0x7f33ec0ffa40 2024-04-06T02:47:49.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.204+0000 7f33eaffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33ec105a40 0x7f33ec106ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:49.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.204+0000 7f33eaffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33ec105a40 0x7f33ec106ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:32948/0 (socket says 172.21.15.105:32948) 2024-04-06T02:47:49.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.204+0000 7f33eaffd700 1 -- 172.21.15.105:0/2322509118 learned_addr learned my addr 172.21.15.105:0/2322509118 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:47:49.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.204+0000 7f33eaffd700 1 -- 172.21.15.105:0/2322509118 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33ec0ffa40 msgr2=0x7f33ec1069b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:49.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.204+0000 7f33eaffd700 1 --2- 172.21.15.105:0/2322509118 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33ec0ffa40 0x7f33ec1069b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:49.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.204+0000 7f33eaffd700 1 -- 172.21.15.105:0/2322509118 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d40095f0 con 0x7f33ec105a40 2024-04-06T02:47:49.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.205+0000 7f33eaffd700 1 --2- 172.21.15.105:0/2322509118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33ec105a40 0x7f33ec106ef0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f33d4006010 tx=0x7f33d400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:49.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.205+0000 7f33e8ff9700 1 -- 172.21.15.105:0/2322509118 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33d401d070 con 0x7f33ec105a40 2024-04-06T02:47:49.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.205+0000 7f33e8ff9700 1 -- 172.21.15.105:0/2322509118 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f33d400f460 con 0x7f33ec105a40 2024-04-06T02:47:49.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.205+0000 7f33f1f4f700 1 -- 172.21.15.105:0/2322509118 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33ec1077f0 con 0x7f33ec105a40 2024-04-06T02:47:49.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.205+0000 7f33e8ff9700 1 -- 172.21.15.105:0/2322509118 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33d4021640 con 0x7f33ec105a40 2024-04-06T02:47:49.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.205+0000 7f33f1f4f700 1 -- 172.21.15.105:0/2322509118 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33ec1a2cc0 con 0x7f33ec105a40 2024-04-06T02:47:49.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.205+0000 7f33f1f4f700 1 -- 172.21.15.105:0/2322509118 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f33ec10cfc0 con 0x7f33ec105a40 2024-04-06T02:47:49.194 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.206+0000 7f33e8ff9700 1 -- 172.21.15.105:0/2322509118 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f33d400faf0 con 0x7f33ec105a40 2024-04-06T02:47:49.194 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.206+0000 7f33e8ff9700 1 --2- 172.21.15.105:0/2322509118 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f33d806c4e0 0x7f33d806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:49.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.207+0000 7f33e8ff9700 1 -- 172.21.15.105:0/2322509118 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f33d408c830 con 0x7f33ec105a40 2024-04-06T02:47:49.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.207+0000 7f33eb7fe700 1 --2- 172.21.15.105:0/2322509118 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f33d806c4e0 0x7f33d806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:49.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.207+0000 7f33e8ff9700 1 --2- 172.21.15.105:0/2322509118 >> [v2:172.21.15.105:6818/3651690266,v1:172.21.15.105:6819/3651690266] conn(0x7f33d80721c0 0x7f33d80745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:49.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.207+0000 7f33e8ff9700 1 -- 172.21.15.105:0/2322509118 --> [v2:172.21.15.105:6818/3651690266,v1:172.21.15.105:6819/3651690266] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f33d8074c90 con 0x7f33d80721c0 2024-04-06T02:47:49.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.207+0000 7f33e8ff9700 1 -- 172.21.15.105:0/2322509118 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33d40912a0 con 0x7f33ec105a40 2024-04-06T02:47:49.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.207+0000 7f33ebfff700 1 --2- 172.21.15.105:0/2322509118 >> [v2:172.21.15.105:6818/3651690266,v1:172.21.15.105:6819/3651690266] conn(0x7f33d80721c0 0x7f33d80745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:49.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.207+0000 7f33eb7fe700 1 --2- 172.21.15.105:0/2322509118 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f33d806c4e0 0x7f33d806e9a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f33e400ab90 tx=0x7f33e4009250 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:49.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.207+0000 7f33ebfff700 1 --2- 172.21.15.105:0/2322509118 >> [v2:172.21.15.105:6818/3651690266,v1:172.21.15.105:6819/3651690266] conn(0x7f33d80721c0 0x7f33d80745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:49.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.208+0000 7f33e8ff9700 1 -- 172.21.15.105:0/2322509118 <== osd.2 v2:172.21.15.105:6818/3651690266 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f33d8074c90 con 0x7f33d80721c0 2024-04-06T02:47:49.208 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.223+0000 7f33f1f4f700 1 -- 172.21.15.105:0/2322509118 --> [v2:172.21.15.105:6818/3651690266,v1:172.21.15.105:6819/3651690266] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f33ec04ea90 con 0x7f33d80721c0 2024-04-06T02:47:49.209 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.223+0000 7f33e8ff9700 1 -- 172.21.15.105:0/2322509118 <== osd.2 v2:172.21.15.105:6818/3651690266 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f33ec04ea90 con 0x7f33d80721c0 2024-04-06T02:47:49.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.223+0000 7f33f1f4f700 1 -- 172.21.15.105:0/2322509118 >> [v2:172.21.15.105:6818/3651690266,v1:172.21.15.105:6819/3651690266] conn(0x7f33d80721c0 msgr2=0x7f33d80745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:49.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.223+0000 7f33f1f4f700 1 --2- 172.21.15.105:0/2322509118 >> [v2:172.21.15.105:6818/3651690266,v1:172.21.15.105:6819/3651690266] conn(0x7f33d80721c0 0x7f33d80745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:49.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.223+0000 7f33f1f4f700 1 -- 172.21.15.105:0/2322509118 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f33d806c4e0 msgr2=0x7f33d806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:49.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.223+0000 7f33f1f4f700 1 --2- 172.21.15.105:0/2322509118 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f33d806c4e0 0x7f33d806e9a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f33e400ab90 tx=0x7f33e4009250 comp rx=0 tx=0).stop 2024-04-06T02:47:49.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.224+0000 7f33f1f4f700 1 -- 172.21.15.105:0/2322509118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33ec105a40 msgr2=0x7f33ec106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:49.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.224+0000 7f33f1f4f700 1 --2- 172.21.15.105:0/2322509118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33ec105a40 0x7f33ec106ef0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f33d4006010 tx=0x7f33d400be20 comp rx=0 tx=0).stop 2024-04-06T02:47:49.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.224+0000 7f33f1f4f700 1 -- 172.21.15.105:0/2322509118 shutdown_connections 2024-04-06T02:47:49.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.224+0000 7f33f1f4f700 1 --2- 172.21.15.105:0/2322509118 >> [v2:172.21.15.105:6818/3651690266,v1:172.21.15.105:6819/3651690266] conn(0x7f33d80721c0 0x7f33d80745e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:49.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.224+0000 7f33f1f4f700 1 --2- 172.21.15.105:0/2322509118 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f33d806c4e0 0x7f33d806e9a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:49.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.224+0000 7f33f1f4f700 1 --2- 172.21.15.105:0/2322509118 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f33ec0ffa40 0x7f33ec1069b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:49.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.224+0000 7f33f1f4f700 1 --2- 172.21.15.105:0/2322509118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33ec105a40 0x7f33ec106ef0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:49.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.224+0000 7f33f1f4f700 1 -- 172.21.15.105:0/2322509118 >> 172.21.15.105:0/2322509118 conn(0x7f33ec0fb570 msgr2=0x7f33ec0fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:49.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.224+0000 7f33f1f4f700 1 -- 172.21.15.105:0/2322509118 shutdown_connections 2024-04-06T02:47:49.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:49.224+0000 7f33f1f4f700 1 -- 172.21.15.105:0/2322509118 wait complete. 2024-04-06T02:47:49.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:49 smithi105 ceph-mon[29228]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:49.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:49 smithi184 ceph-mon[37439]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:51.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:51 smithi105 ceph-mon[29228]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:51.963 INFO:teuthology.orchestra.run.smithi105.stdout:81604378647 2024-04-06T02:47:51.964 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.2 2024-04-06T02:47:51.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:51 smithi184 ceph-mon[37439]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:53.564 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:53 smithi105 ceph-mon[29228]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:53.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:53 smithi184 ceph-mon[37439]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:54.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.724+0000 7f52a1243700 1 -- 172.21.15.105:0/1184232466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f529c105a40 msgr2=0x7f529c105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:54.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.724+0000 7f52a1243700 1 --2- 172.21.15.105:0/1184232466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f529c105a40 0x7f529c105e20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f528c009910 tx=0x7f528c009c20 comp rx=0 tx=0).stop 2024-04-06T02:47:54.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.724+0000 7f52a1243700 1 -- 172.21.15.105:0/1184232466 shutdown_connections 2024-04-06T02:47:54.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.724+0000 7f52a1243700 1 --2- 172.21.15.105:0/1184232466 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f529c0ffbd0 0x7f529c100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:54.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.724+0000 7f52a1243700 1 --2- 172.21.15.105:0/1184232466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f529c105a40 0x7f529c105e20 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:54.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.724+0000 7f52a1243700 1 -- 172.21.15.105:0/1184232466 >> 172.21.15.105:0/1184232466 conn(0x7f529c0755a0 msgr2=0x7f529c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:54.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.724+0000 7f52a1243700 1 -- 172.21.15.105:0/1184232466 shutdown_connections 2024-04-06T02:47:54.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.724+0000 7f52a1243700 1 -- 172.21.15.105:0/1184232466 wait complete. 2024-04-06T02:47:54.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.725+0000 7f52a1243700 1 Processor -- start 2024-04-06T02:47:54.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.725+0000 7f52a1243700 1 -- start start 2024-04-06T02:47:54.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.725+0000 7f52a1243700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f529c0ffbd0 0x7f529c19c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:54.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.725+0000 7f52a1243700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f529c105a40 0x7f529c19cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:54.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.725+0000 7f52a1243700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529c19d140 con 0x7f529c105a40 2024-04-06T02:47:54.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.725+0000 7f52a1243700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529c1965f0 con 0x7f529c0ffbd0 2024-04-06T02:47:54.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.725+0000 7f5293fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f529c105a40 0x7f529c19cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:54.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.725+0000 7f5293fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f529c105a40 0x7f529c19cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60840/0 (socket says 172.21.15.105:60840) 2024-04-06T02:47:54.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.725+0000 7f5293fff700 1 -- 172.21.15.105:0/1104962139 learned_addr learned my addr 172.21.15.105:0/1104962139 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:47:54.713 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.726+0000 7f529ad9d700 1 --2- 172.21.15.105:0/1104962139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f529c0ffbd0 0x7f529c19c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:54.713 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.726+0000 7f5293fff700 1 -- 172.21.15.105:0/1104962139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f529c0ffbd0 msgr2=0x7f529c19c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:54.713 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.726+0000 7f5293fff700 1 --2- 172.21.15.105:0/1104962139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f529c0ffbd0 0x7f529c19c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:54.713 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.726+0000 7f5293fff700 1 -- 172.21.15.105:0/1104962139 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528c0095f0 con 0x7f529c105a40 2024-04-06T02:47:54.713 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.726+0000 7f5293fff700 1 --2- 172.21.15.105:0/1104962139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f529c105a40 0x7f529c19cab0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f528400b4b0 tx=0x7f528400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:54.713 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.726+0000 7f5298d99700 1 -- 172.21.15.105:0/1104962139 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5284011800 con 0x7f529c105a40 2024-04-06T02:47:54.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.726+0000 7f52a1243700 1 -- 172.21.15.105:0/1104962139 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f529c1968d0 con 0x7f529c105a40 2024-04-06T02:47:54.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.726+0000 7f5298d99700 1 -- 172.21.15.105:0/1104962139 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5284011e10 con 0x7f529c105a40 2024-04-06T02:47:54.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.726+0000 7f5298d99700 1 -- 172.21.15.105:0/1104962139 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f528400f2e0 con 0x7f529c105a40 2024-04-06T02:47:54.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.727+0000 7f52a1243700 1 -- 172.21.15.105:0/1104962139 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f529c196e20 con 0x7f529c105a40 2024-04-06T02:47:54.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.727+0000 7f52a1243700 1 -- 172.21.15.105:0/1104962139 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f529c10cfc0 con 0x7f529c105a40 2024-04-06T02:47:54.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.727+0000 7f5298d99700 1 -- 172.21.15.105:0/1104962139 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f5284011960 con 0x7f529c105a40 2024-04-06T02:47:54.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.728+0000 7f5298d99700 1 --2- 172.21.15.105:0/1104962139 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f528806c2d0 0x7f528806e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:54.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.728+0000 7f5298d99700 1 -- 172.21.15.105:0/1104962139 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f528408b0d0 con 0x7f529c105a40 2024-04-06T02:47:54.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.728+0000 7f529ad9d700 1 --2- 172.21.15.105:0/1104962139 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f528806c2d0 0x7f528806e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:54.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.728+0000 7f5298d99700 1 --2- 172.21.15.105:0/1104962139 >> [v2:172.21.15.184:6808/2237051966,v1:172.21.15.184:6809/2237051966] conn(0x7f5288071e40 0x7f5288074260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:54.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.728+0000 7f5298d99700 1 -- 172.21.15.105:0/1104962139 --> [v2:172.21.15.184:6808/2237051966,v1:172.21.15.184:6809/2237051966] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5288074910 con 0x7f5288071e40 2024-04-06T02:47:54.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.728+0000 7f5298d99700 1 -- 172.21.15.105:0/1104962139 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f528408d7a0 con 0x7f529c105a40 2024-04-06T02:47:54.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.728+0000 7f529ad9d700 1 --2- 172.21.15.105:0/1104962139 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f528806c2d0 0x7f528806e790 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f528c0051a0 tx=0x7f528c005fb0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:54.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.728+0000 7f529b59e700 1 --2- 172.21.15.105:0/1104962139 >> [v2:172.21.15.184:6808/2237051966,v1:172.21.15.184:6809/2237051966] conn(0x7f5288071e40 0x7f5288074260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:54.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.729+0000 7f529b59e700 1 --2- 172.21.15.105:0/1104962139 >> [v2:172.21.15.184:6808/2237051966,v1:172.21.15.184:6809/2237051966] conn(0x7f5288071e40 0x7f5288074260 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:54.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.730+0000 7f5298d99700 1 -- 172.21.15.105:0/1104962139 <== osd.4 v2:172.21.15.184:6808/2237051966 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f5288074910 con 0x7f5288071e40 2024-04-06T02:47:54.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.745+0000 7f52a1243700 1 -- 172.21.15.105:0/1104962139 --> [v2:172.21.15.184:6808/2237051966,v1:172.21.15.184:6809/2237051966] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f529c04ea90 con 0x7f5288071e40 2024-04-06T02:47:54.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.745+0000 7f5298d99700 1 -- 172.21.15.105:0/1104962139 <== osd.4 v2:172.21.15.184:6808/2237051966 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f529c04ea90 con 0x7f5288071e40 2024-04-06T02:47:54.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.746+0000 7f52a1243700 1 -- 172.21.15.105:0/1104962139 >> [v2:172.21.15.184:6808/2237051966,v1:172.21.15.184:6809/2237051966] conn(0x7f5288071e40 msgr2=0x7f5288074260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:54.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.746+0000 7f52a1243700 1 --2- 172.21.15.105:0/1104962139 >> [v2:172.21.15.184:6808/2237051966,v1:172.21.15.184:6809/2237051966] conn(0x7f5288071e40 0x7f5288074260 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:54.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.746+0000 7f52a1243700 1 -- 172.21.15.105:0/1104962139 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f528806c2d0 msgr2=0x7f528806e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:54.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.746+0000 7f52a1243700 1 --2- 172.21.15.105:0/1104962139 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f528806c2d0 0x7f528806e790 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f528c0051a0 tx=0x7f528c005fb0 comp rx=0 tx=0).stop 2024-04-06T02:47:54.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.746+0000 7f52a1243700 1 -- 172.21.15.105:0/1104962139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f529c105a40 msgr2=0x7f529c19cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:54.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.746+0000 7f52a1243700 1 --2- 172.21.15.105:0/1104962139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f529c105a40 0x7f529c19cab0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f528400b4b0 tx=0x7f528400b870 comp rx=0 tx=0).stop 2024-04-06T02:47:54.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.746+0000 7f52a1243700 1 -- 172.21.15.105:0/1104962139 shutdown_connections 2024-04-06T02:47:54.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.746+0000 7f52a1243700 1 --2- 172.21.15.105:0/1104962139 >> [v2:172.21.15.184:6808/2237051966,v1:172.21.15.184:6809/2237051966] conn(0x7f5288071e40 0x7f5288074260 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:54.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.746+0000 7f52a1243700 1 --2- 172.21.15.105:0/1104962139 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f528806c2d0 0x7f528806e790 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:54.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.746+0000 7f52a1243700 1 --2- 172.21.15.105:0/1104962139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f529c0ffbd0 0x7f529c19c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:54.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.746+0000 7f52a1243700 1 --2- 172.21.15.105:0/1104962139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f529c105a40 0x7f529c19cab0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:54.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.746+0000 7f52a1243700 1 -- 172.21.15.105:0/1104962139 >> 172.21.15.105:0/1104962139 conn(0x7f529c0755a0 msgr2=0x7f529c068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:54.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.746+0000 7f52a1243700 1 -- 172.21.15.105:0/1104962139 shutdown_connections 2024-04-06T02:47:54.733 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.746+0000 7f52a1243700 1 -- 172.21.15.105:0/1104962139 wait complete. 2024-04-06T02:47:54.842 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:54 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:47:54.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:54 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:47:55.317 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.332+0000 7efd8c105700 1 -- 172.21.15.105:0/1364222634 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd84069020 msgr2=0x7efd84105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:55.317 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.332+0000 7efd8c105700 1 --2- 172.21.15.105:0/1364222634 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd84069020 0x7efd84105790 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7efd74009910 tx=0x7efd74009c20 comp rx=0 tx=0).stop 2024-04-06T02:47:55.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.332+0000 7efd8c105700 1 -- 172.21.15.105:0/1364222634 shutdown_connections 2024-04-06T02:47:55.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.332+0000 7efd8c105700 1 --2- 172.21.15.105:0/1364222634 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd84069020 0x7efd84105790 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:55.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.332+0000 7efd8c105700 1 --2- 172.21.15.105:0/1364222634 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd84068670 0x7efd84068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:55.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.332+0000 7efd8c105700 1 -- 172.21.15.105:0/1364222634 >> 172.21.15.105:0/1364222634 conn(0x7efd840755a0 msgr2=0x7efd840759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:55.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.332+0000 7efd8c105700 1 -- 172.21.15.105:0/1364222634 shutdown_connections 2024-04-06T02:47:55.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.332+0000 7efd8c105700 1 -- 172.21.15.105:0/1364222634 wait complete. 2024-04-06T02:47:55.320 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.332+0000 7efd8c105700 1 Processor -- start 2024-04-06T02:47:55.320 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.332+0000 7efd8c105700 1 -- start start 2024-04-06T02:47:55.320 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.333+0000 7efd8c105700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd84068670 0x7efd84196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:55.320 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.333+0000 7efd8c105700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd84069020 0x7efd84196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:55.320 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.333+0000 7efd8c105700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd841972d0 con 0x7efd84069020 2024-04-06T02:47:55.320 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.333+0000 7efd8c105700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8419a000 con 0x7efd84068670 2024-04-06T02:47:55.320 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.333+0000 7efd896a0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd84069020 0x7efd84196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:55.320 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.333+0000 7efd896a0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd84069020 0x7efd84196c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60892/0 (socket says 172.21.15.105:60892) 2024-04-06T02:47:55.320 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.333+0000 7efd896a0700 1 -- 172.21.15.105:0/1283483015 learned_addr learned my addr 172.21.15.105:0/1283483015 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:47:55.320 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.333+0000 7efd89ea1700 1 --2- 172.21.15.105:0/1283483015 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd84068670 0x7efd84196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:55.321 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.333+0000 7efd896a0700 1 -- 172.21.15.105:0/1283483015 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd84068670 msgr2=0x7efd84196700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:55.321 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.333+0000 7efd896a0700 1 --2- 172.21.15.105:0/1283483015 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd84068670 0x7efd84196700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:55.321 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.333+0000 7efd896a0700 1 -- 172.21.15.105:0/1283483015 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd740095f0 con 0x7efd84069020 2024-04-06T02:47:55.321 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.333+0000 7efd896a0700 1 --2- 172.21.15.105:0/1283483015 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd84069020 0x7efd84196c40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7efd740092e0 tx=0x7efd7400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:55.321 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.333+0000 7efd7affd700 1 -- 172.21.15.105:0/1283483015 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd7401d070 con 0x7efd84069020 2024-04-06T02:47:55.321 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.333+0000 7efd8c105700 1 -- 172.21.15.105:0/1283483015 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd8419a280 con 0x7efd84069020 2024-04-06T02:47:55.321 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.333+0000 7efd7affd700 1 -- 172.21.15.105:0/1283483015 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7efd74005dc0 con 0x7efd84069020 2024-04-06T02:47:55.321 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.333+0000 7efd7affd700 1 -- 172.21.15.105:0/1283483015 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd74017400 con 0x7efd84069020 2024-04-06T02:47:55.321 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.334+0000 7efd8c105700 1 -- 172.21.15.105:0/1283483015 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd8419a770 con 0x7efd84069020 2024-04-06T02:47:55.321 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.334+0000 7efd8c105700 1 -- 172.21.15.105:0/1283483015 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efd8410e060 con 0x7efd84069020 2024-04-06T02:47:55.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.335+0000 7efd7affd700 1 -- 172.21.15.105:0/1283483015 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7efd7400f460 con 0x7efd84069020 2024-04-06T02:47:55.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.335+0000 7efd7affd700 1 --2- 172.21.15.105:0/1283483015 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7efd7006c530 0x7efd7006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:55.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.335+0000 7efd7affd700 1 -- 172.21.15.105:0/1283483015 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7efd7408cde0 con 0x7efd84069020 2024-04-06T02:47:55.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.335+0000 7efd89ea1700 1 --2- 172.21.15.105:0/1283483015 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7efd7006c530 0x7efd7006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:55.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.336+0000 7efd7affd700 1 --2- 172.21.15.105:0/1283483015 >> [v2:172.21.15.184:6816/1371998009,v1:172.21.15.184:6817/1371998009] conn(0x7efd70072210 0x7efd70074630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:55.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.336+0000 7efd7affd700 1 -- 172.21.15.105:0/1283483015 --> [v2:172.21.15.184:6816/1371998009,v1:172.21.15.184:6817/1371998009] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd70074ce0 con 0x7efd70072210 2024-04-06T02:47:55.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.336+0000 7efd7affd700 1 -- 172.21.15.105:0/1283483015 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd7408d190 con 0x7efd84069020 2024-04-06T02:47:55.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.336+0000 7efd8a6a2700 1 --2- 172.21.15.105:0/1283483015 >> [v2:172.21.15.184:6816/1371998009,v1:172.21.15.184:6817/1371998009] conn(0x7efd70072210 0x7efd70074630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:55.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.336+0000 7efd89ea1700 1 --2- 172.21.15.105:0/1283483015 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7efd7006c530 0x7efd7006e9f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efd7c005950 tx=0x7efd7c00b410 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:55.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.336+0000 7efd8a6a2700 1 --2- 172.21.15.105:0/1283483015 >> [v2:172.21.15.184:6816/1371998009,v1:172.21.15.184:6817/1371998009] conn(0x7efd70072210 0x7efd70074630 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:55.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.337+0000 7efd7affd700 1 -- 172.21.15.105:0/1283483015 <== osd.5 v2:172.21.15.184:6816/1371998009 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7efd70074ce0 con 0x7efd70072210 2024-04-06T02:47:55.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.352+0000 7efd8c105700 1 -- 172.21.15.105:0/1283483015 --> [v2:172.21.15.184:6816/1371998009,v1:172.21.15.184:6817/1371998009] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efd8404ea90 con 0x7efd70072210 2024-04-06T02:47:55.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.353+0000 7efd7affd700 1 -- 172.21.15.105:0/1283483015 <== osd.5 v2:172.21.15.184:6816/1371998009 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7efd8404ea90 con 0x7efd70072210 2024-04-06T02:47:55.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.353+0000 7efd8c105700 1 -- 172.21.15.105:0/1283483015 >> [v2:172.21.15.184:6816/1371998009,v1:172.21.15.184:6817/1371998009] conn(0x7efd70072210 msgr2=0x7efd70074630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:55.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.353+0000 7efd8c105700 1 --2- 172.21.15.105:0/1283483015 >> [v2:172.21.15.184:6816/1371998009,v1:172.21.15.184:6817/1371998009] conn(0x7efd70072210 0x7efd70074630 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:55.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.353+0000 7efd8c105700 1 -- 172.21.15.105:0/1283483015 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7efd7006c530 msgr2=0x7efd7006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:55.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.353+0000 7efd8c105700 1 --2- 172.21.15.105:0/1283483015 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7efd7006c530 0x7efd7006e9f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efd7c005950 tx=0x7efd7c00b410 comp rx=0 tx=0).stop 2024-04-06T02:47:55.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.353+0000 7efd8c105700 1 -- 172.21.15.105:0/1283483015 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd84069020 msgr2=0x7efd84196c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:55.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.353+0000 7efd8c105700 1 --2- 172.21.15.105:0/1283483015 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd84069020 0x7efd84196c40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7efd740092e0 tx=0x7efd7400bd70 comp rx=0 tx=0).stop 2024-04-06T02:47:55.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.353+0000 7efd8c105700 1 -- 172.21.15.105:0/1283483015 shutdown_connections 2024-04-06T02:47:55.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.353+0000 7efd8c105700 1 --2- 172.21.15.105:0/1283483015 >> [v2:172.21.15.184:6816/1371998009,v1:172.21.15.184:6817/1371998009] conn(0x7efd70072210 0x7efd70074630 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:55.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.353+0000 7efd8c105700 1 --2- 172.21.15.105:0/1283483015 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7efd7006c530 0x7efd7006e9f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:55.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.353+0000 7efd8c105700 1 --2- 172.21.15.105:0/1283483015 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd84068670 0x7efd84196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:55.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.353+0000 7efd8c105700 1 --2- 172.21.15.105:0/1283483015 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd84069020 0x7efd84196c40 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:55.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.353+0000 7efd8c105700 1 -- 172.21.15.105:0/1283483015 >> 172.21.15.105:0/1283483015 conn(0x7efd840755a0 msgr2=0x7efd84105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:55.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.353+0000 7efd8c105700 1 -- 172.21.15.105:0/1283483015 shutdown_connections 2024-04-06T02:47:55.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:55.353+0000 7efd8c105700 1 -- 172.21.15.105:0/1283483015 wait complete. 2024-04-06T02:47:55.681 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.956+0000 7fc508692700 1 -- 172.21.15.105:0/3131302421 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc500101780 msgr2=0x7fc5001057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:55.681 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.956+0000 7fc508692700 1 --2- 172.21.15.105:0/3131302421 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc500101780 0x7fc5001057d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc009910 tx=0x7fc4fc009c20 comp rx=0 tx=0).stop 2024-04-06T02:47:55.681 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.956+0000 7fc508692700 1 -- 172.21.15.105:0/3131302421 shutdown_connections 2024-04-06T02:47:55.681 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.956+0000 7fc508692700 1 --2- 172.21.15.105:0/3131302421 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc500101780 0x7fc5001057d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:55.681 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.956+0000 7fc508692700 1 --2- 172.21.15.105:0/3131302421 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc500100dd0 0x7fc5001011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:55.681 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.956+0000 7fc508692700 1 -- 172.21.15.105:0/3131302421 >> 172.21.15.105:0/3131302421 conn(0x7fc5000fc640 msgr2=0x7fc5000fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:55.681 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.956+0000 7fc508692700 1 -- 172.21.15.105:0/3131302421 shutdown_connections 2024-04-06T02:47:55.681 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.956+0000 7fc508692700 1 -- 172.21.15.105:0/3131302421 wait complete. 2024-04-06T02:47:55.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.957+0000 7fc508692700 1 Processor -- start 2024-04-06T02:47:55.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.957+0000 7fc508692700 1 -- start start 2024-04-06T02:47:55.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.957+0000 7fc508692700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc500100dd0 0x7fc50019c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:55.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.957+0000 7fc508692700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc500101780 0x7fc50019cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:55.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.957+0000 7fc508692700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc50019d160 con 0x7fc500101780 2024-04-06T02:47:55.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.957+0000 7fc508692700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc500196610 con 0x7fc500100dd0 2024-04-06T02:47:55.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.957+0000 7fc505c2d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc500101780 0x7fc50019cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:55.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.957+0000 7fc505c2d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc500101780 0x7fc50019cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60872/0 (socket says 172.21.15.105:60872) 2024-04-06T02:47:55.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.957+0000 7fc505c2d700 1 -- 172.21.15.105:0/3002469982 learned_addr learned my addr 172.21.15.105:0/3002469982 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:47:55.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.958+0000 7fc50642e700 1 --2- 172.21.15.105:0/3002469982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc500100dd0 0x7fc50019c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:55.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.958+0000 7fc505c2d700 1 -- 172.21.15.105:0/3002469982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc500100dd0 msgr2=0x7fc50019c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:55.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.958+0000 7fc505c2d700 1 --2- 172.21.15.105:0/3002469982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc500100dd0 0x7fc50019c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:55.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.958+0000 7fc505c2d700 1 -- 172.21.15.105:0/3002469982 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4fc0095f0 con 0x7fc500101780 2024-04-06T02:47:55.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.958+0000 7fc505c2d700 1 --2- 172.21.15.105:0/3002469982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc500101780 0x7fc50019cad0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc009fd0 tx=0x7fc4fc00fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:55.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.958+0000 7fc4f77fe700 1 -- 172.21.15.105:0/3002469982 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4fc01d070 con 0x7fc500101780 2024-04-06T02:47:55.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.958+0000 7fc508692700 1 -- 172.21.15.105:0/3002469982 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc500196890 con 0x7fc500101780 2024-04-06T02:47:55.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.958+0000 7fc4f77fe700 1 -- 172.21.15.105:0/3002469982 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc4fc005640 con 0x7fc500101780 2024-04-06T02:47:55.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.958+0000 7fc4f77fe700 1 -- 172.21.15.105:0/3002469982 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4fc0226a0 con 0x7fc500101780 2024-04-06T02:47:55.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.958+0000 7fc508692700 1 -- 172.21.15.105:0/3002469982 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc500196d80 con 0x7fc500101780 2024-04-06T02:47:55.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.959+0000 7fc508692700 1 -- 172.21.15.105:0/3002469982 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc50010dff0 con 0x7fc500101780 2024-04-06T02:47:55.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.959+0000 7fc4f77fe700 1 -- 172.21.15.105:0/3002469982 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fc4fc003680 con 0x7fc500101780 2024-04-06T02:47:55.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.960+0000 7fc4f77fe700 1 --2- 172.21.15.105:0/3002469982 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fc4ec06c4e0 0x7fc4ec06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:55.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.960+0000 7fc50642e700 1 --2- 172.21.15.105:0/3002469982 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fc4ec06c4e0 0x7fc4ec06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:55.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.960+0000 7fc4f77fe700 1 -- 172.21.15.105:0/3002469982 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fc4fc08c970 con 0x7fc500101780 2024-04-06T02:47:55.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.960+0000 7fc4f77fe700 1 --2- 172.21.15.105:0/3002469982 >> [v2:172.21.15.184:6800/330017535,v1:172.21.15.184:6801/330017535] conn(0x7fc4ec0721c0 0x7fc4ec0745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:55.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.960+0000 7fc4f77fe700 1 -- 172.21.15.105:0/3002469982 --> [v2:172.21.15.184:6800/330017535,v1:172.21.15.184:6801/330017535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc4ec074c90 con 0x7fc4ec0721c0 2024-04-06T02:47:55.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.960+0000 7fc4f77fe700 1 -- 172.21.15.105:0/3002469982 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4fc0912a0 con 0x7fc500101780 2024-04-06T02:47:55.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.960+0000 7fc50642e700 1 --2- 172.21.15.105:0/3002469982 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fc4ec06c4e0 0x7fc4ec06e9a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0009b30 tx=0x7fc4f0009260 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:55.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.960+0000 7fc506c2f700 1 --2- 172.21.15.105:0/3002469982 >> [v2:172.21.15.184:6800/330017535,v1:172.21.15.184:6801/330017535] conn(0x7fc4ec0721c0 0x7fc4ec0745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:55.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.960+0000 7fc506c2f700 1 --2- 172.21.15.105:0/3002469982 >> [v2:172.21.15.184:6800/330017535,v1:172.21.15.184:6801/330017535] conn(0x7fc4ec0721c0 0x7fc4ec0745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:55.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.961+0000 7fc4f77fe700 1 -- 172.21.15.105:0/3002469982 <== osd.3 v2:172.21.15.184:6800/330017535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fc4ec074c90 con 0x7fc4ec0721c0 2024-04-06T02:47:55.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.976+0000 7fc508692700 1 -- 172.21.15.105:0/3002469982 --> [v2:172.21.15.184:6800/330017535,v1:172.21.15.184:6801/330017535] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc50004ea90 con 0x7fc4ec0721c0 2024-04-06T02:47:55.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.977+0000 7fc4f77fe700 1 -- 172.21.15.105:0/3002469982 <== osd.3 v2:172.21.15.184:6800/330017535 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc50004ea90 con 0x7fc4ec0721c0 2024-04-06T02:47:55.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.977+0000 7fc508692700 1 -- 172.21.15.105:0/3002469982 >> [v2:172.21.15.184:6800/330017535,v1:172.21.15.184:6801/330017535] conn(0x7fc4ec0721c0 msgr2=0x7fc4ec0745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:55.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.977+0000 7fc508692700 1 --2- 172.21.15.105:0/3002469982 >> [v2:172.21.15.184:6800/330017535,v1:172.21.15.184:6801/330017535] conn(0x7fc4ec0721c0 0x7fc4ec0745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:55.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.978+0000 7fc508692700 1 -- 172.21.15.105:0/3002469982 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fc4ec06c4e0 msgr2=0x7fc4ec06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:55.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.978+0000 7fc508692700 1 --2- 172.21.15.105:0/3002469982 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fc4ec06c4e0 0x7fc4ec06e9a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0009b30 tx=0x7fc4f0009260 comp rx=0 tx=0).stop 2024-04-06T02:47:55.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.978+0000 7fc508692700 1 -- 172.21.15.105:0/3002469982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc500101780 msgr2=0x7fc50019cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:55.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.978+0000 7fc508692700 1 --2- 172.21.15.105:0/3002469982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc500101780 0x7fc50019cad0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc009fd0 tx=0x7fc4fc00fab0 comp rx=0 tx=0).stop 2024-04-06T02:47:55.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.978+0000 7fc508692700 1 -- 172.21.15.105:0/3002469982 shutdown_connections 2024-04-06T02:47:55.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.978+0000 7fc508692700 1 --2- 172.21.15.105:0/3002469982 >> [v2:172.21.15.184:6800/330017535,v1:172.21.15.184:6801/330017535] conn(0x7fc4ec0721c0 0x7fc4ec0745e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:55.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.978+0000 7fc508692700 1 --2- 172.21.15.105:0/3002469982 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fc4ec06c4e0 0x7fc4ec06e9a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:55.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.978+0000 7fc508692700 1 --2- 172.21.15.105:0/3002469982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc500100dd0 0x7fc50019c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:55.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.978+0000 7fc508692700 1 --2- 172.21.15.105:0/3002469982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc500101780 0x7fc50019cad0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:55.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.978+0000 7fc508692700 1 -- 172.21.15.105:0/3002469982 >> 172.21.15.105:0/3002469982 conn(0x7fc5000fc640 msgr2=0x7fc5000fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:55.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.978+0000 7fc508692700 1 -- 172.21.15.105:0/3002469982 shutdown_connections 2024-04-06T02:47:55.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:54.978+0000 7fc508692700 1 -- 172.21.15.105:0/3002469982 wait complete. 2024-04-06T02:47:55.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:55 smithi105 ceph-mon[29228]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:55.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:55 smithi184 ceph-mon[37439]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:56.590 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.603+0000 7f02258d5700 1 -- 172.21.15.105:0/1995099796 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02200ffbd0 msgr2=0x7f0220100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:56.590 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.603+0000 7f02258d5700 1 --2- 172.21.15.105:0/1995099796 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02200ffbd0 0x7f0220100050 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0208009910 tx=0x7f0208009c20 comp rx=0 tx=0).stop 2024-04-06T02:47:56.590 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.603+0000 7f02258d5700 1 -- 172.21.15.105:0/1995099796 shutdown_connections 2024-04-06T02:47:56.591 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.603+0000 7f02258d5700 1 --2- 172.21.15.105:0/1995099796 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02200ffbd0 0x7f0220100050 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:56.591 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.603+0000 7f02258d5700 1 --2- 172.21.15.105:0/1995099796 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0220105a40 0x7f0220105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:56.591 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.603+0000 7f02258d5700 1 -- 172.21.15.105:0/1995099796 >> 172.21.15.105:0/1995099796 conn(0x7f02200755a0 msgr2=0x7f02200759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:56.591 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.603+0000 7f02258d5700 1 -- 172.21.15.105:0/1995099796 shutdown_connections 2024-04-06T02:47:56.591 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.603+0000 7f02258d5700 1 -- 172.21.15.105:0/1995099796 wait complete. 2024-04-06T02:47:56.591 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.603+0000 7f02258d5700 1 Processor -- start 2024-04-06T02:47:56.591 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.604+0000 7f02258d5700 1 -- start start 2024-04-06T02:47:56.591 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.604+0000 7f02258d5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02200ffbd0 0x7f022019c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:56.591 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.604+0000 7f02258d5700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0220105a40 0x7f022019cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:56.591 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.604+0000 7f02258d5700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022019d140 con 0x7f02200ffbd0 2024-04-06T02:47:56.591 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.604+0000 7f02258d5700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02201965f0 con 0x7f0220105a40 2024-04-06T02:47:56.592 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.604+0000 7f021effd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02200ffbd0 0x7f022019c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:56.592 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.604+0000 7f021effd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02200ffbd0 0x7f022019c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60904/0 (socket says 172.21.15.105:60904) 2024-04-06T02:47:56.592 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.604+0000 7f021effd700 1 -- 172.21.15.105:0/1410068730 learned_addr learned my addr 172.21.15.105:0/1410068730 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:47:56.592 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.604+0000 7f02167fc700 1 --2- 172.21.15.105:0/1410068730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0220105a40 0x7f022019cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:56.592 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.604+0000 7f021effd700 1 -- 172.21.15.105:0/1410068730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0220105a40 msgr2=0x7f022019cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:56.592 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.604+0000 7f021effd700 1 --2- 172.21.15.105:0/1410068730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0220105a40 0x7f022019cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:56.592 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.604+0000 7f021effd700 1 -- 172.21.15.105:0/1410068730 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02080095f0 con 0x7f02200ffbd0 2024-04-06T02:47:56.592 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.605+0000 7f021effd700 1 --2- 172.21.15.105:0/1410068730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02200ffbd0 0x7f022019c570 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f021000b4b0 tx=0x7f021000b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:56.592 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.605+0000 7f02248d3700 1 -- 172.21.15.105:0/1410068730 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0210011800 con 0x7f02200ffbd0 2024-04-06T02:47:56.592 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.605+0000 7f02258d5700 1 -- 172.21.15.105:0/1410068730 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02201968d0 con 0x7f02200ffbd0 2024-04-06T02:47:56.593 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.605+0000 7f02248d3700 1 -- 172.21.15.105:0/1410068730 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0210011e10 con 0x7f02200ffbd0 2024-04-06T02:47:56.594 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.605+0000 7f02248d3700 1 -- 172.21.15.105:0/1410068730 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f021000f2e0 con 0x7f02200ffbd0 2024-04-06T02:47:56.594 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.605+0000 7f02258d5700 1 -- 172.21.15.105:0/1410068730 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0220196e20 con 0x7f02200ffbd0 2024-04-06T02:47:56.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.606+0000 7f02258d5700 1 -- 172.21.15.105:0/1410068730 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f022010cfc0 con 0x7f02200ffbd0 2024-04-06T02:47:56.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.606+0000 7f02248d3700 1 -- 172.21.15.105:0/1410068730 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f0210011960 con 0x7f02200ffbd0 2024-04-06T02:47:56.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.607+0000 7f02248d3700 1 --2- 172.21.15.105:0/1410068730 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f020c06c4e0 0x7f020c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:56.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.607+0000 7f02167fc700 1 --2- 172.21.15.105:0/1410068730 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f020c06c4e0 0x7f020c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:56.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.607+0000 7f02248d3700 1 -- 172.21.15.105:0/1410068730 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f021008b0e0 con 0x7f02200ffbd0 2024-04-06T02:47:56.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.607+0000 7f02248d3700 1 --2- 172.21.15.105:0/1410068730 >> [v2:172.21.15.105:6810/1345191629,v1:172.21.15.105:6811/1345191629] conn(0x7f020c0721c0 0x7f020c0745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:56.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.607+0000 7f02248d3700 1 -- 172.21.15.105:0/1410068730 --> [v2:172.21.15.105:6810/1345191629,v1:172.21.15.105:6811/1345191629] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f020c074c90 con 0x7f020c0721c0 2024-04-06T02:47:56.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.607+0000 7f02248d3700 1 -- 172.21.15.105:0/1410068730 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f021008d7b0 con 0x7f02200ffbd0 2024-04-06T02:47:56.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.607+0000 7f021f7fe700 1 --2- 172.21.15.105:0/1410068730 >> [v2:172.21.15.105:6810/1345191629,v1:172.21.15.105:6811/1345191629] conn(0x7f020c0721c0 0x7f020c0745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:56.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.607+0000 7f02167fc700 1 --2- 172.21.15.105:0/1410068730 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f020c06c4e0 0x7f020c06e9a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0220104d20 tx=0x7f020801a040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:56.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.607+0000 7f021f7fe700 1 --2- 172.21.15.105:0/1410068730 >> [v2:172.21.15.105:6810/1345191629,v1:172.21.15.105:6811/1345191629] conn(0x7f020c0721c0 0x7f020c0745e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:56.596 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.608+0000 7f02248d3700 1 -- 172.21.15.105:0/1410068730 <== osd.1 v2:172.21.15.105:6810/1345191629 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f020c074c90 con 0x7f020c0721c0 2024-04-06T02:47:56.614 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.628+0000 7f02258d5700 1 -- 172.21.15.105:0/1410068730 --> [v2:172.21.15.105:6810/1345191629,v1:172.21.15.105:6811/1345191629] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f022004ea90 con 0x7f020c0721c0 2024-04-06T02:47:56.614 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.629+0000 7f02248d3700 1 -- 172.21.15.105:0/1410068730 <== osd.1 v2:172.21.15.105:6810/1345191629 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f022004ea90 con 0x7f020c0721c0 2024-04-06T02:47:56.615 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.629+0000 7f02258d5700 1 -- 172.21.15.105:0/1410068730 >> [v2:172.21.15.105:6810/1345191629,v1:172.21.15.105:6811/1345191629] conn(0x7f020c0721c0 msgr2=0x7f020c0745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:56.615 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.629+0000 7f02258d5700 1 --2- 172.21.15.105:0/1410068730 >> [v2:172.21.15.105:6810/1345191629,v1:172.21.15.105:6811/1345191629] conn(0x7f020c0721c0 0x7f020c0745e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:56.615 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.629+0000 7f02258d5700 1 -- 172.21.15.105:0/1410068730 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f020c06c4e0 msgr2=0x7f020c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:56.615 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.629+0000 7f02258d5700 1 --2- 172.21.15.105:0/1410068730 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f020c06c4e0 0x7f020c06e9a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0220104d20 tx=0x7f020801a040 comp rx=0 tx=0).stop 2024-04-06T02:47:56.615 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.629+0000 7f02258d5700 1 -- 172.21.15.105:0/1410068730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02200ffbd0 msgr2=0x7f022019c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:56.615 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.629+0000 7f02258d5700 1 --2- 172.21.15.105:0/1410068730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02200ffbd0 0x7f022019c570 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f021000b4b0 tx=0x7f021000b7c0 comp rx=0 tx=0).stop 2024-04-06T02:47:56.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.629+0000 7f02258d5700 1 -- 172.21.15.105:0/1410068730 shutdown_connections 2024-04-06T02:47:56.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.629+0000 7f02258d5700 1 --2- 172.21.15.105:0/1410068730 >> [v2:172.21.15.105:6810/1345191629,v1:172.21.15.105:6811/1345191629] conn(0x7f020c0721c0 0x7f020c0745e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:56.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.629+0000 7f02258d5700 1 --2- 172.21.15.105:0/1410068730 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f020c06c4e0 0x7f020c06e9a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:56.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.629+0000 7f02258d5700 1 --2- 172.21.15.105:0/1410068730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02200ffbd0 0x7f022019c570 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:56.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.629+0000 7f02258d5700 1 --2- 172.21.15.105:0/1410068730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0220105a40 0x7f022019cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:56.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.629+0000 7f02258d5700 1 -- 172.21.15.105:0/1410068730 >> 172.21.15.105:0/1410068730 conn(0x7f02200755a0 msgr2=0x7f0220068610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:56.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.629+0000 7f02258d5700 1 -- 172.21.15.105:0/1410068730 shutdown_connections 2024-04-06T02:47:56.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:56.629+0000 7f02258d5700 1 -- 172.21.15.105:0/1410068730 wait complete. 2024-04-06T02:47:57.282 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.295+0000 7f48ea903700 1 -- 172.21.15.105:0/3232942438 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f48e4105a40 msgr2=0x7f48e4105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:57.282 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.295+0000 7f48ea903700 1 --2- 172.21.15.105:0/3232942438 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f48e4105a40 0x7f48e4105e20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f48d4009910 tx=0x7f48d4009c20 comp rx=0 tx=0).stop 2024-04-06T02:47:57.282 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.295+0000 7f48ea903700 1 -- 172.21.15.105:0/3232942438 shutdown_connections 2024-04-06T02:47:57.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.295+0000 7f48ea903700 1 --2- 172.21.15.105:0/3232942438 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48e40ffbd0 0x7f48e4100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:57.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.295+0000 7f48ea903700 1 --2- 172.21.15.105:0/3232942438 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f48e4105a40 0x7f48e4105e20 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:57.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.295+0000 7f48ea903700 1 -- 172.21.15.105:0/3232942438 >> 172.21.15.105:0/3232942438 conn(0x7f48e40755a0 msgr2=0x7f48e40759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:57.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.295+0000 7f48ea903700 1 -- 172.21.15.105:0/3232942438 shutdown_connections 2024-04-06T02:47:57.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.295+0000 7f48ea903700 1 -- 172.21.15.105:0/3232942438 wait complete. 2024-04-06T02:47:57.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.295+0000 7f48ea903700 1 Processor -- start 2024-04-06T02:47:57.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.295+0000 7f48ea903700 1 -- start start 2024-04-06T02:47:57.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.296+0000 7f48ea903700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f48e40ffbd0 0x7f48e41975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:57.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.296+0000 7f48ea903700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48e4105a40 0x7f48e4197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:57.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.296+0000 7f48ea903700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e4198110 con 0x7f48e40ffbd0 2024-04-06T02:47:57.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.296+0000 7f48ea903700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e4198250 con 0x7f48e4105a40 2024-04-06T02:47:57.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.296+0000 7f48e3fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f48e40ffbd0 0x7f48e41975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:57.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.296+0000 7f48e3fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f48e40ffbd0 0x7f48e41975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60922/0 (socket says 172.21.15.105:60922) 2024-04-06T02:47:57.284 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.296+0000 7f48e3fff700 1 -- 172.21.15.105:0/3035040956 learned_addr learned my addr 172.21.15.105:0/3035040956 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:47:57.284 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.296+0000 7f48e37fe700 1 --2- 172.21.15.105:0/3035040956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48e4105a40 0x7f48e4197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:57.284 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.296+0000 7f48e3fff700 1 -- 172.21.15.105:0/3035040956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48e4105a40 msgr2=0x7f48e4197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:57.284 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.296+0000 7f48e3fff700 1 --2- 172.21.15.105:0/3035040956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48e4105a40 0x7f48e4197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:57.284 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.296+0000 7f48e3fff700 1 -- 172.21.15.105:0/3035040956 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d40095f0 con 0x7f48e40ffbd0 2024-04-06T02:47:57.284 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.296+0000 7f48e3fff700 1 --2- 172.21.15.105:0/3035040956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f48e40ffbd0 0x7f48e41975b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f48d4006010 tx=0x7f48d400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:57.284 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.297+0000 7f48e17fa700 1 -- 172.21.15.105:0/3035040956 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48d401d070 con 0x7f48e40ffbd0 2024-04-06T02:47:57.284 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.297+0000 7f48ea903700 1 -- 172.21.15.105:0/3035040956 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48e41a0620 con 0x7f48e40ffbd0 2024-04-06T02:47:57.284 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.297+0000 7f48e17fa700 1 -- 172.21.15.105:0/3035040956 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f48d400f460 con 0x7f48e40ffbd0 2024-04-06T02:47:57.284 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.297+0000 7f48e17fa700 1 -- 172.21.15.105:0/3035040956 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48d4017630 con 0x7f48e40ffbd0 2024-04-06T02:47:57.284 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.297+0000 7f48ea903700 1 -- 172.21.15.105:0/3035040956 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48e41a0a30 con 0x7f48e40ffbd0 2024-04-06T02:47:57.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.297+0000 7f48ea903700 1 -- 172.21.15.105:0/3035040956 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f48e410cf40 con 0x7f48e40ffbd0 2024-04-06T02:47:57.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.298+0000 7f48e17fa700 1 -- 172.21.15.105:0/3035040956 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f48d400f5d0 con 0x7f48e40ffbd0 2024-04-06T02:47:57.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.299+0000 7f48e17fa700 1 --2- 172.21.15.105:0/3035040956 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f48cc06c4e0 0x7f48cc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:57.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.299+0000 7f48e17fa700 1 -- 172.21.15.105:0/3035040956 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f48d408c940 con 0x7f48e40ffbd0 2024-04-06T02:47:57.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.299+0000 7f48e37fe700 1 --2- 172.21.15.105:0/3035040956 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f48cc06c4e0 0x7f48cc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:57.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.299+0000 7f48e17fa700 1 --2- 172.21.15.105:0/3035040956 >> [v2:172.21.15.105:6802/2018109658,v1:172.21.15.105:6803/2018109658] conn(0x7f48cc0721c0 0x7f48cc0745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:47:57.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.299+0000 7f48e17fa700 1 -- 172.21.15.105:0/3035040956 --> [v2:172.21.15.105:6802/2018109658,v1:172.21.15.105:6803/2018109658] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f48cc074c90 con 0x7f48cc0721c0 2024-04-06T02:47:57.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.299+0000 7f48e17fa700 1 -- 172.21.15.105:0/3035040956 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48d40912a0 con 0x7f48e40ffbd0 2024-04-06T02:47:57.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.299+0000 7f48e8ea0700 1 --2- 172.21.15.105:0/3035040956 >> [v2:172.21.15.105:6802/2018109658,v1:172.21.15.105:6803/2018109658] conn(0x7f48cc0721c0 0x7f48cc0745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:47:57.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.299+0000 7f48e37fe700 1 --2- 172.21.15.105:0/3035040956 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f48cc06c4e0 0x7f48cc06e9a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f48e4068cc0 tx=0x7f48d800b410 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:57.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.299+0000 7f48e8ea0700 1 --2- 172.21.15.105:0/3035040956 >> [v2:172.21.15.105:6802/2018109658,v1:172.21.15.105:6803/2018109658] conn(0x7f48cc0721c0 0x7f48cc0745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:47:57.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.300+0000 7f48e17fa700 1 -- 172.21.15.105:0/3035040956 <== osd.0 v2:172.21.15.105:6802/2018109658 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f48cc074c90 con 0x7f48cc0721c0 2024-04-06T02:47:57.300 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.315+0000 7f48ea903700 1 -- 172.21.15.105:0/3035040956 --> [v2:172.21.15.105:6802/2018109658,v1:172.21.15.105:6803/2018109658] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f48e404ea90 con 0x7f48cc0721c0 2024-04-06T02:47:57.301 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.315+0000 7f48e17fa700 1 -- 172.21.15.105:0/3035040956 <== osd.0 v2:172.21.15.105:6802/2018109658 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f48e404ea90 con 0x7f48cc0721c0 2024-04-06T02:47:57.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.316+0000 7f48ea903700 1 -- 172.21.15.105:0/3035040956 >> [v2:172.21.15.105:6802/2018109658,v1:172.21.15.105:6803/2018109658] conn(0x7f48cc0721c0 msgr2=0x7f48cc0745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:57.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.316+0000 7f48ea903700 1 --2- 172.21.15.105:0/3035040956 >> [v2:172.21.15.105:6802/2018109658,v1:172.21.15.105:6803/2018109658] conn(0x7f48cc0721c0 0x7f48cc0745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:57.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.316+0000 7f48ea903700 1 -- 172.21.15.105:0/3035040956 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f48cc06c4e0 msgr2=0x7f48cc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:57.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.316+0000 7f48ea903700 1 --2- 172.21.15.105:0/3035040956 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f48cc06c4e0 0x7f48cc06e9a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f48e4068cc0 tx=0x7f48d800b410 comp rx=0 tx=0).stop 2024-04-06T02:47:57.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.316+0000 7f48ea903700 1 -- 172.21.15.105:0/3035040956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f48e40ffbd0 msgr2=0x7f48e41975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:47:57.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.316+0000 7f48ea903700 1 --2- 172.21.15.105:0/3035040956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f48e40ffbd0 0x7f48e41975b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f48d4006010 tx=0x7f48d400bd70 comp rx=0 tx=0).stop 2024-04-06T02:47:57.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.316+0000 7f48ea903700 1 -- 172.21.15.105:0/3035040956 shutdown_connections 2024-04-06T02:47:57.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.316+0000 7f48ea903700 1 --2- 172.21.15.105:0/3035040956 >> [v2:172.21.15.105:6802/2018109658,v1:172.21.15.105:6803/2018109658] conn(0x7f48cc0721c0 0x7f48cc0745e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:57.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.316+0000 7f48ea903700 1 --2- 172.21.15.105:0/3035040956 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f48cc06c4e0 0x7f48cc06e9a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:57.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.316+0000 7f48ea903700 1 --2- 172.21.15.105:0/3035040956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f48e40ffbd0 0x7f48e41975b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:57.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.316+0000 7f48ea903700 1 --2- 172.21.15.105:0/3035040956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f48e4105a40 0x7f48e4197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:47:57.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.316+0000 7f48ea903700 1 -- 172.21.15.105:0/3035040956 >> 172.21.15.105:0/3035040956 conn(0x7f48e40755a0 msgr2=0x7f48e40fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:47:57.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.316+0000 7f48ea903700 1 -- 172.21.15.105:0/3035040956 shutdown_connections 2024-04-06T02:47:57.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:47:57.316+0000 7f48ea903700 1 -- 172.21.15.105:0/3035040956 wait complete. 2024-04-06T02:47:57.911 INFO:teuthology.orchestra.run.smithi105.stdout:115964117011 2024-04-06T02:47:57.911 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.3 2024-04-06T02:47:57.926 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:57 smithi105 ceph-mon[29228]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:57.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:57 smithi184 ceph-mon[37439]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:58.418 INFO:teuthology.orchestra.run.smithi105.stdout:150323855373 2024-04-06T02:47:58.419 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.4 2024-04-06T02:47:59.124 INFO:teuthology.orchestra.run.smithi105.stdout:38654705700 2024-04-06T02:47:59.124 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.0 2024-04-06T02:47:59.611 INFO:teuthology.orchestra.run.smithi105.stdout:60129542173 2024-04-06T02:47:59.611 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.1 2024-04-06T02:47:59.655 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:47:59.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:47:59 smithi105 ceph-mon[29228]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:47:59.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:47:59 smithi184 ceph-mon[37439]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:00.054 INFO:teuthology.orchestra.run.smithi105.stdout:171798691849 2024-04-06T02:48:00.054 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.5 2024-04-06T02:48:00.132 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:48:01.413 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:48:01.435 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:48:01.896 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:01 smithi105 ceph-mon[29228]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:01.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:01 smithi184 ceph-mon[37439]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:02.404 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:48:02.411 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:48:03.641 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:03 smithi105 ceph-mon[29228]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:03.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:03 smithi184 ceph-mon[37439]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:03.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.953+0000 7f1f45106700 1 -- 172.21.15.105:0/2821521219 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f40105a50 msgr2=0x7f1f40105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:03.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.953+0000 7f1f45106700 1 --2- 172.21.15.105:0/2821521219 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f40105a50 0x7f1f40105e30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1f30009910 tx=0x7f1f30009c20 comp rx=0 tx=0).stop 2024-04-06T02:48:03.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.953+0000 7f1f45106700 1 -- 172.21.15.105:0/2821521219 shutdown_connections 2024-04-06T02:48:03.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.953+0000 7f1f45106700 1 --2- 172.21.15.105:0/2821521219 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f400ffa50 0x7f1f400ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:03.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.953+0000 7f1f45106700 1 --2- 172.21.15.105:0/2821521219 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f40105a50 0x7f1f40105e30 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:03.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.953+0000 7f1f45106700 1 -- 172.21.15.105:0/2821521219 >> 172.21.15.105:0/2821521219 conn(0x7f1f400fb540 msgr2=0x7f1f400fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:03.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.954+0000 7f1f45106700 1 -- 172.21.15.105:0/2821521219 shutdown_connections 2024-04-06T02:48:03.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.954+0000 7f1f45106700 1 -- 172.21.15.105:0/2821521219 wait complete. 2024-04-06T02:48:03.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.954+0000 7f1f45106700 1 Processor -- start 2024-04-06T02:48:03.942 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.954+0000 7f1f45106700 1 -- start start 2024-04-06T02:48:03.942 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.955+0000 7f1f45106700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f400ffa50 0x7f1f401975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:03.942 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.955+0000 7f1f45106700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f40105a50 0x7f1f40197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:03.942 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.955+0000 7f1f45106700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f40198120 con 0x7f1f40105a50 2024-04-06T02:48:03.942 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.955+0000 7f1f45106700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f40198260 con 0x7f1f400ffa50 2024-04-06T02:48:03.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.955+0000 7f1f3e59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f40105a50 0x7f1f40197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:03.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.955+0000 7f1f3e59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f40105a50 0x7f1f40197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:59664/0 (socket says 172.21.15.105:59664) 2024-04-06T02:48:03.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.955+0000 7f1f3e59c700 1 -- 172.21.15.105:0/3164060858 learned_addr learned my addr 172.21.15.105:0/3164060858 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:48:03.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.955+0000 7f1f3ed9d700 1 --2- 172.21.15.105:0/3164060858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f400ffa50 0x7f1f401975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:03.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.955+0000 7f1f3e59c700 1 -- 172.21.15.105:0/3164060858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f400ffa50 msgr2=0x7f1f401975c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:03.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.955+0000 7f1f3e59c700 1 --2- 172.21.15.105:0/3164060858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f400ffa50 0x7f1f401975c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:03.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.955+0000 7f1f3e59c700 1 -- 172.21.15.105:0/3164060858 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f300095f0 con 0x7f1f40105a50 2024-04-06T02:48:03.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.955+0000 7f1f3e59c700 1 --2- 172.21.15.105:0/3164060858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f40105a50 0x7f1f40197b00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1f3400d8d0 tx=0x7f1f3400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:03.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.956+0000 7f1f27fff700 1 -- 172.21.15.105:0/3164060858 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f340098a0 con 0x7f1f40105a50 2024-04-06T02:48:03.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.956+0000 7f1f45106700 1 -- 172.21.15.105:0/3164060858 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f401a0630 con 0x7f1f40105a50 2024-04-06T02:48:03.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.956+0000 7f1f27fff700 1 -- 172.21.15.105:0/3164060858 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1f34010460 con 0x7f1f40105a50 2024-04-06T02:48:03.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.956+0000 7f1f27fff700 1 -- 172.21.15.105:0/3164060858 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f3400f5f0 con 0x7f1f40105a50 2024-04-06T02:48:03.944 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.956+0000 7f1f45106700 1 -- 172.21.15.105:0/3164060858 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f401a0af0 con 0x7f1f40105a50 2024-04-06T02:48:03.944 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.957+0000 7f1f45106700 1 -- 172.21.15.105:0/3164060858 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f40108050 con 0x7f1f40105a50 2024-04-06T02:48:03.944 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.958+0000 7f1f27fff700 1 -- 172.21.15.105:0/3164060858 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f1f34009a00 con 0x7f1f40105a50 2024-04-06T02:48:03.945 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.958+0000 7f1f27fff700 1 --2- 172.21.15.105:0/3164060858 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f1f2806c4e0 0x7f1f2806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:03.945 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.958+0000 7f1f27fff700 1 -- 172.21.15.105:0/3164060858 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f1f3408b3d0 con 0x7f1f40105a50 2024-04-06T02:48:03.945 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.958+0000 7f1f3ed9d700 1 --2- 172.21.15.105:0/3164060858 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f1f2806c4e0 0x7f1f2806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:03.945 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.959+0000 7f1f3ed9d700 1 --2- 172.21.15.105:0/3164060858 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f1f2806c4e0 0x7f1f2806e9a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1f30006010 tx=0x7f1f3001a040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:03.952 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.967+0000 7f1f27fff700 1 -- 172.21.15.105:0/3164060858 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1f34055ef0 con 0x7f1f40105a50 2024-04-06T02:48:04.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.880+0000 7f1708e83700 1 -- 172.21.15.105:0/2932216426 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1704069020 msgr2=0x7f1704105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:04.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.880+0000 7f1708e83700 1 --2- 172.21.15.105:0/2932216426 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1704069020 0x7f1704105790 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f16f8009910 tx=0x7f16f8009c20 comp rx=0 tx=0).stop 2024-04-06T02:48:04.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.880+0000 7f1708e83700 1 -- 172.21.15.105:0/2932216426 shutdown_connections 2024-04-06T02:48:04.037 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.880+0000 7f1708e83700 1 --2- 172.21.15.105:0/2932216426 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1704069020 0x7f1704105790 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:04.037 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.880+0000 7f1708e83700 1 --2- 172.21.15.105:0/2932216426 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1704068670 0x7f1704068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:04.037 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.880+0000 7f1708e83700 1 -- 172.21.15.105:0/2932216426 >> 172.21.15.105:0/2932216426 conn(0x7f17040755a0 msgr2=0x7f17040759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:04.037 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.880+0000 7f1708e83700 1 -- 172.21.15.105:0/2932216426 shutdown_connections 2024-04-06T02:48:04.037 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.880+0000 7f1708e83700 1 -- 172.21.15.105:0/2932216426 wait complete. 2024-04-06T02:48:04.037 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.881+0000 7f1708e83700 1 Processor -- start 2024-04-06T02:48:04.037 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.881+0000 7f1708e83700 1 -- start start 2024-04-06T02:48:04.037 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.881+0000 7f1708e83700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1704068670 0x7f1704196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:04.037 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.881+0000 7f1708e83700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1704069020 0x7f1704196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:04.037 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.881+0000 7f1708e83700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17041972d0 con 0x7f1704069020 2024-04-06T02:48:04.037 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.881+0000 7f1708e83700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170419a000 con 0x7f1704068670 2024-04-06T02:48:04.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.882+0000 7f1701d9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1704069020 0x7f1704196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:04.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.882+0000 7f1701d9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1704069020 0x7f1704196c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:59650/0 (socket says 172.21.15.105:59650) 2024-04-06T02:48:04.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.882+0000 7f1701d9b700 1 -- 172.21.15.105:0/3773656167 learned_addr learned my addr 172.21.15.105:0/3773656167 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:48:04.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.882+0000 7f170259c700 1 --2- 172.21.15.105:0/3773656167 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1704068670 0x7f1704196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:04.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.882+0000 7f1701d9b700 1 -- 172.21.15.105:0/3773656167 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1704068670 msgr2=0x7f1704196700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:04.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.882+0000 7f1701d9b700 1 --2- 172.21.15.105:0/3773656167 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1704068670 0x7f1704196700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:04.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.882+0000 7f1701d9b700 1 -- 172.21.15.105:0/3773656167 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f80095f0 con 0x7f1704069020 2024-04-06T02:48:04.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.882+0000 7f1701d9b700 1 --2- 172.21.15.105:0/3773656167 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1704069020 0x7f1704196c40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f16f80092e0 tx=0x7f16f800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:04.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.882+0000 7f16f37fe700 1 -- 172.21.15.105:0/3773656167 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16f801d070 con 0x7f1704069020 2024-04-06T02:48:04.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.882+0000 7f1708e83700 1 -- 172.21.15.105:0/3773656167 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f170419a280 con 0x7f1704069020 2024-04-06T02:48:04.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.882+0000 7f16f37fe700 1 -- 172.21.15.105:0/3773656167 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f16f800f460 con 0x7f1704069020 2024-04-06T02:48:04.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.882+0000 7f16f37fe700 1 -- 172.21.15.105:0/3773656167 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16f8017630 con 0x7f1704069020 2024-04-06T02:48:04.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.882+0000 7f1708e83700 1 -- 172.21.15.105:0/3773656167 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f170419a770 con 0x7f1704069020 2024-04-06T02:48:04.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.883+0000 7f1708e83700 1 -- 172.21.15.105:0/3773656167 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17041090a0 con 0x7f1704069020 2024-04-06T02:48:04.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.884+0000 7f16f37fe700 1 -- 172.21.15.105:0/3773656167 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f16f8017860 con 0x7f1704069020 2024-04-06T02:48:04.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.884+0000 7f16f37fe700 1 --2- 172.21.15.105:0/3773656167 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f16ec06c4e0 0x7f16ec06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:04.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.884+0000 7f16f37fe700 1 -- 172.21.15.105:0/3773656167 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f16f808c970 con 0x7f1704069020 2024-04-06T02:48:04.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.884+0000 7f170259c700 1 --2- 172.21.15.105:0/3773656167 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f16ec06c4e0 0x7f16ec06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:04.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.884+0000 7f170259c700 1 --2- 172.21.15.105:0/3773656167 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f16ec06c4e0 0x7f16ec06e9a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f16f4005950 tx=0x7f16f40058e0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:04.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:03.889+0000 7f16f37fe700 1 -- 172.21.15.105:0/3773656167 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f16f8057490 con 0x7f1704069020 2024-04-06T02:48:04.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:04.092+0000 7f1708e83700 1 -- 172.21.15.105:0/3773656167 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f170402cd30 con 0x7f1704069020 2024-04-06T02:48:04.078 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:04.092+0000 7f16f37fe700 1 -- 172.21.15.105:0/3773656167 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f16f805aab0 con 0x7f1704069020 2024-04-06T02:48:04.078 INFO:teuthology.orchestra.run.smithi105.stdout:81604378650 2024-04-06T02:48:04.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:04.094+0000 7f1708e83700 1 -- 172.21.15.105:0/3773656167 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f16ec06c4e0 msgr2=0x7f16ec06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:04.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:04.094+0000 7f1708e83700 1 --2- 172.21.15.105:0/3773656167 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f16ec06c4e0 0x7f16ec06e9a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f16f4005950 tx=0x7f16f40058e0 comp rx=0 tx=0).stop 2024-04-06T02:48:04.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:04.094+0000 7f1708e83700 1 -- 172.21.15.105:0/3773656167 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1704069020 msgr2=0x7f1704196c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:04.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:04.094+0000 7f1708e83700 1 --2- 172.21.15.105:0/3773656167 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1704069020 0x7f1704196c40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f16f80092e0 tx=0x7f16f800be20 comp rx=0 tx=0).stop 2024-04-06T02:48:04.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:04.094+0000 7f1708e83700 1 -- 172.21.15.105:0/3773656167 shutdown_connections 2024-04-06T02:48:04.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:04.094+0000 7f1708e83700 1 --2- 172.21.15.105:0/3773656167 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f16ec06c4e0 0x7f16ec06e9a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:04.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:04.094+0000 7f1708e83700 1 --2- 172.21.15.105:0/3773656167 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1704068670 0x7f1704196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:04.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:04.094+0000 7f1708e83700 1 --2- 172.21.15.105:0/3773656167 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1704069020 0x7f1704196c40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:04.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:04.094+0000 7f1708e83700 1 -- 172.21.15.105:0/3773656167 >> 172.21.15.105:0/3773656167 conn(0x7f17040755a0 msgr2=0x7f1704105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:04.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:04.094+0000 7f1708e83700 1 -- 172.21.15.105:0/3773656167 shutdown_connections 2024-04-06T02:48:04.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:04.094+0000 7f1708e83700 1 -- 172.21.15.105:0/3773656167 wait complete. 2024-04-06T02:48:04.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:04.171+0000 7f1f45106700 1 -- 172.21.15.105:0/3164060858 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f1f4004ea90 con 0x7f1f40105a50 2024-04-06T02:48:04.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:04.172+0000 7f1f27fff700 1 -- 172.21.15.105:0/3164060858 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f1f34059510 con 0x7f1f40105a50 2024-04-06T02:48:04.158 INFO:teuthology.orchestra.run.smithi105.stdout:150323855375 2024-04-06T02:48:04.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:04.174+0000 7f1f45106700 1 -- 172.21.15.105:0/3164060858 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f1f2806c4e0 msgr2=0x7f1f2806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:04.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:04.174+0000 7f1f45106700 1 --2- 172.21.15.105:0/3164060858 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f1f2806c4e0 0x7f1f2806e9a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1f30006010 tx=0x7f1f3001a040 comp rx=0 tx=0).stop 2024-04-06T02:48:04.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:04.174+0000 7f1f45106700 1 -- 172.21.15.105:0/3164060858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f40105a50 msgr2=0x7f1f40197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:04.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:04.174+0000 7f1f45106700 1 --2- 172.21.15.105:0/3164060858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f40105a50 0x7f1f40197b00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1f3400d8d0 tx=0x7f1f3400dbe0 comp rx=0 tx=0).stop 2024-04-06T02:48:04.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:04.174+0000 7f1f45106700 1 -- 172.21.15.105:0/3164060858 shutdown_connections 2024-04-06T02:48:04.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:04.174+0000 7f1f45106700 1 --2- 172.21.15.105:0/3164060858 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f1f2806c4e0 0x7f1f2806e9a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:04.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:04.174+0000 7f1f45106700 1 --2- 172.21.15.105:0/3164060858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f400ffa50 0x7f1f401975c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:04.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:04.174+0000 7f1f45106700 1 --2- 172.21.15.105:0/3164060858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f40105a50 0x7f1f40197b00 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:04.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:04.174+0000 7f1f45106700 1 -- 172.21.15.105:0/3164060858 >> 172.21.15.105:0/3164060858 conn(0x7f1f400fb540 msgr2=0x7f1f400fbf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:04.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:04.174+0000 7f1f45106700 1 -- 172.21.15.105:0/3164060858 shutdown_connections 2024-04-06T02:48:04.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:04.174+0000 7f1f45106700 1 -- 172.21.15.105:0/3164060858 wait complete. 2024-04-06T02:48:04.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:04 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/3773656167' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-06T02:48:04.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:04 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/3164060858' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-06T02:48:04.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:04 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/3773656167' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-06T02:48:04.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:04 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/3164060858' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-06T02:48:05.623 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:05 smithi105 ceph-mon[29228]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:05.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:05 smithi184 ceph-mon[37439]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:05.933 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378647 got 81604378650 for osd.2 2024-04-06T02:48:05.934 DEBUG:teuthology.parallel:result is None 2024-04-06T02:48:07.096 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855373 got 150323855375 for osd.4 2024-04-06T02:48:07.096 DEBUG:teuthology.parallel:result is None 2024-04-06T02:48:07.719 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:07 smithi105 ceph-mon[29228]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:07.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.833+0000 7f338f70a700 1 -- 172.21.15.105:0/158543298 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3388101770 msgr2=0x7f33881057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:07.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.833+0000 7f338f70a700 1 --2- 172.21.15.105:0/158543298 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3388101770 0x7f33881057c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3378009910 tx=0x7f3378009c20 comp rx=0 tx=0).stop 2024-04-06T02:48:07.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.833+0000 7f338f70a700 1 -- 172.21.15.105:0/158543298 shutdown_connections 2024-04-06T02:48:07.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.833+0000 7f338f70a700 1 --2- 172.21.15.105:0/158543298 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3388101770 0x7f33881057c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:07.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.833+0000 7f338f70a700 1 --2- 172.21.15.105:0/158543298 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3388100dc0 0x7f33881011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.833+0000 7f338f70a700 1 -- 172.21.15.105:0/158543298 >> 172.21.15.105:0/158543298 conn(0x7f33880fc650 msgr2=0x7f33880fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.833+0000 7f338f70a700 1 -- 172.21.15.105:0/158543298 shutdown_connections 2024-04-06T02:48:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.833+0000 7f338f70a700 1 -- 172.21.15.105:0/158543298 wait complete. 2024-04-06T02:48:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.834+0000 7f338f70a700 1 Processor -- start 2024-04-06T02:48:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.834+0000 7f338f70a700 1 -- start start 2024-04-06T02:48:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.834+0000 7f338f70a700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3388100dc0 0x7f3388198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.834+0000 7f338f70a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3388101770 0x7f3388198eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.834+0000 7f338f70a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3388199540 con 0x7f3388101770 2024-04-06T02:48:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.834+0000 7f338f70a700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338819d350 con 0x7f3388100dc0 2024-04-06T02:48:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.834+0000 7f338cca5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3388101770 0x7f3388198eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.834+0000 7f338cca5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3388101770 0x7f3388198eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:59674/0 (socket says 172.21.15.105:59674) 2024-04-06T02:48:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.834+0000 7f338cca5700 1 -- 172.21.15.105:0/453333687 learned_addr learned my addr 172.21.15.105:0/453333687 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:48:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.834+0000 7f338d4a6700 1 --2- 172.21.15.105:0/453333687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3388100dc0 0x7f3388198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.834+0000 7f338cca5700 1 -- 172.21.15.105:0/453333687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3388100dc0 msgr2=0x7f3388198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:07.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.834+0000 7f338cca5700 1 --2- 172.21.15.105:0/453333687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3388100dc0 0x7f3388198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:07.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.834+0000 7f338cca5700 1 -- 172.21.15.105:0/453333687 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33780095f0 con 0x7f3388101770 2024-04-06T02:48:07.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.835+0000 7f338cca5700 1 --2- 172.21.15.105:0/453333687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3388101770 0x7f3388198eb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f33780092e0 tx=0x7f337800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:07.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.835+0000 7f337e7fc700 1 -- 172.21.15.105:0/453333687 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f337801d070 con 0x7f3388101770 2024-04-06T02:48:07.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.835+0000 7f337e7fc700 1 -- 172.21.15.105:0/453333687 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3378004fa0 con 0x7f3388101770 2024-04-06T02:48:07.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.835+0000 7f338f70a700 1 -- 172.21.15.105:0/453333687 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f338819d5d0 con 0x7f3388101770 2024-04-06T02:48:07.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.835+0000 7f337e7fc700 1 -- 172.21.15.105:0/453333687 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f337800f690 con 0x7f3388101770 2024-04-06T02:48:07.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.835+0000 7f338f70a700 1 -- 172.21.15.105:0/453333687 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f338819dac0 con 0x7f3388101770 2024-04-06T02:48:07.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.836+0000 7f338f70a700 1 -- 172.21.15.105:0/453333687 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3388104f40 con 0x7f3388101770 2024-04-06T02:48:07.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.836+0000 7f337e7fc700 1 -- 172.21.15.105:0/453333687 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f3378022470 con 0x7f3388101770 2024-04-06T02:48:07.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.837+0000 7f337e7fc700 1 --2- 172.21.15.105:0/453333687 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f337406c4e0 0x7f337406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:07.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.837+0000 7f338d4a6700 1 --2- 172.21.15.105:0/453333687 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f337406c4e0 0x7f337406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:07.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.837+0000 7f337e7fc700 1 -- 172.21.15.105:0/453333687 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f337808cbe0 con 0x7f3388101770 2024-04-06T02:48:07.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.837+0000 7f338d4a6700 1 --2- 172.21.15.105:0/453333687 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f337406c4e0 0x7f337406e9a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3384005950 tx=0x7f338400a300 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:07.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:07.841+0000 7f337e7fc700 1 -- 172.21.15.105:0/453333687 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f337805b2b0 con 0x7f3388101770 2024-04-06T02:48:07.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:07 smithi184 ceph-mon[37439]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:08.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:08.035+0000 7f338f70a700 1 -- 172.21.15.105:0/453333687 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f338804ea90 con 0x7f3388101770 2024-04-06T02:48:08.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:08.035+0000 7f337e7fc700 1 -- 172.21.15.105:0/453333687 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3378027020 con 0x7f3388101770 2024-04-06T02:48:08.021 INFO:teuthology.orchestra.run.smithi105.stdout:38654705702 2024-04-06T02:48:08.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:08.037+0000 7f338f70a700 1 -- 172.21.15.105:0/453333687 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f337406c4e0 msgr2=0x7f337406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:08.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:08.037+0000 7f338f70a700 1 --2- 172.21.15.105:0/453333687 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f337406c4e0 0x7f337406e9a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3384005950 tx=0x7f338400a300 comp rx=0 tx=0).stop 2024-04-06T02:48:08.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:08.037+0000 7f338f70a700 1 -- 172.21.15.105:0/453333687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3388101770 msgr2=0x7f3388198eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:08.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:08.037+0000 7f338f70a700 1 --2- 172.21.15.105:0/453333687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3388101770 0x7f3388198eb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f33780092e0 tx=0x7f337800bd70 comp rx=0 tx=0).stop 2024-04-06T02:48:08.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:08.038+0000 7f338f70a700 1 -- 172.21.15.105:0/453333687 shutdown_connections 2024-04-06T02:48:08.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:08.038+0000 7f338f70a700 1 --2- 172.21.15.105:0/453333687 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f337406c4e0 0x7f337406e9a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:08.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:08.038+0000 7f338f70a700 1 --2- 172.21.15.105:0/453333687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3388100dc0 0x7f3388198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:08.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:08.038+0000 7f338f70a700 1 --2- 172.21.15.105:0/453333687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3388101770 0x7f3388198eb0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:08.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:08.038+0000 7f338f70a700 1 -- 172.21.15.105:0/453333687 >> 172.21.15.105:0/453333687 conn(0x7f33880fc650 msgr2=0x7f33880fdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:08.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:08.038+0000 7f338f70a700 1 -- 172.21.15.105:0/453333687 shutdown_connections 2024-04-06T02:48:08.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:08.038+0000 7f338f70a700 1 -- 172.21.15.105:0/453333687 wait complete. 2024-04-06T02:48:08.768 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:08 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/453333687' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-06T02:48:08.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:08 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/453333687' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-06T02:48:09.174 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705700 got 38654705702 for osd.0 2024-04-06T02:48:09.175 DEBUG:teuthology.parallel:result is None 2024-04-06T02:48:09.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:09 smithi105 ceph-mon[29228]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:09.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:09 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:48:09.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:09 smithi184 ceph-mon[37439]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:09.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:09 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:48:10.146 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.159+0000 7fc74c679700 1 -- 172.21.15.105:0/1190123405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc744104a50 msgr2=0x7fc744104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:10.146 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.159+0000 7fc74c679700 1 --2- 172.21.15.105:0/1190123405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc744104a50 0x7fc744104e30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc734009910 tx=0x7fc734009c20 comp rx=0 tx=0).stop 2024-04-06T02:48:10.146 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.159+0000 7fc74c679700 1 -- 172.21.15.105:0/1190123405 shutdown_connections 2024-04-06T02:48:10.146 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.159+0000 7fc74c679700 1 --2- 172.21.15.105:0/1190123405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc744100140 0x7fc7441005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:10.146 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.159+0000 7fc74c679700 1 --2- 172.21.15.105:0/1190123405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc744104a50 0x7fc744104e30 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:10.146 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.159+0000 7fc74c679700 1 -- 172.21.15.105:0/1190123405 >> 172.21.15.105:0/1190123405 conn(0x7fc744075e30 msgr2=0x7fc7440fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:10.146 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.159+0000 7fc74c679700 1 -- 172.21.15.105:0/1190123405 shutdown_connections 2024-04-06T02:48:10.147 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.159+0000 7fc74c679700 1 -- 172.21.15.105:0/1190123405 wait complete. 2024-04-06T02:48:10.147 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.159+0000 7fc74c679700 1 Processor -- start 2024-04-06T02:48:10.147 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.159+0000 7fc74c679700 1 -- start start 2024-04-06T02:48:10.147 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.160+0000 7fc74c679700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc744100140 0x7fc7441966e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:10.147 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.160+0000 7fc74c679700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc744104a50 0x7fc744196c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:10.147 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.160+0000 7fc74c679700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7441972b0 con 0x7fc744100140 2024-04-06T02:48:10.147 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.160+0000 7fc74c679700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc744199fe0 con 0x7fc744104a50 2024-04-06T02:48:10.147 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.160+0000 7fc74a415700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc744100140 0x7fc7441966e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:10.147 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.160+0000 7fc74a415700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc744100140 0x7fc7441966e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:59704/0 (socket says 172.21.15.105:59704) 2024-04-06T02:48:10.147 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.160+0000 7fc74a415700 1 -- 172.21.15.105:0/2809768704 learned_addr learned my addr 172.21.15.105:0/2809768704 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:48:10.147 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.160+0000 7fc749c14700 1 --2- 172.21.15.105:0/2809768704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc744104a50 0x7fc744196c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:10.147 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.160+0000 7fc74a415700 1 -- 172.21.15.105:0/2809768704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc744104a50 msgr2=0x7fc744196c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:10.147 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.160+0000 7fc74a415700 1 --2- 172.21.15.105:0/2809768704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc744104a50 0x7fc744196c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:10.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.160+0000 7fc74a415700 1 -- 172.21.15.105:0/2809768704 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7340095f0 con 0x7fc744100140 2024-04-06T02:48:10.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.160+0000 7fc749c14700 1 --2- 172.21.15.105:0/2809768704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc744104a50 0x7fc744196c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T02:48:10.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.160+0000 7fc74a415700 1 --2- 172.21.15.105:0/2809768704 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc744100140 0x7fc7441966e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc734006010 tx=0x7fc73400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:10.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.160+0000 7fc73b7fe700 1 -- 172.21.15.105:0/2809768704 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc73401d070 con 0x7fc744100140 2024-04-06T02:48:10.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.160+0000 7fc73b7fe700 1 -- 172.21.15.105:0/2809768704 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc73400f460 con 0x7fc744100140 2024-04-06T02:48:10.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.160+0000 7fc73b7fe700 1 -- 172.21.15.105:0/2809768704 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc734021640 con 0x7fc744100140 2024-04-06T02:48:10.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.160+0000 7fc74c679700 1 -- 172.21.15.105:0/2809768704 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc74419a260 con 0x7fc744100140 2024-04-06T02:48:10.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.161+0000 7fc74c679700 1 -- 172.21.15.105:0/2809768704 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc74419a7d0 con 0x7fc744100140 2024-04-06T02:48:10.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.162+0000 7fc74c679700 1 -- 172.21.15.105:0/2809768704 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc74404ea90 con 0x7fc744100140 2024-04-06T02:48:10.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.162+0000 7fc73b7fe700 1 -- 172.21.15.105:0/2809768704 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fc73400faf0 con 0x7fc744100140 2024-04-06T02:48:10.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.162+0000 7fc73b7fe700 1 --2- 172.21.15.105:0/2809768704 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fc73006c4e0 0x7fc73006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:10.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.163+0000 7fc749c14700 1 --2- 172.21.15.105:0/2809768704 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fc73006c4e0 0x7fc73006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:10.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.163+0000 7fc73b7fe700 1 -- 172.21.15.105:0/2809768704 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fc73408c870 con 0x7fc744100140 2024-04-06T02:48:10.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.163+0000 7fc749c14700 1 --2- 172.21.15.105:0/2809768704 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fc73006c4e0 0x7fc73006e9a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc744197d00 tx=0x7fc740008040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:10.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.167+0000 7fc73b7fe700 1 -- 172.21.15.105:0/2809768704 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc734057390 con 0x7fc744100140 2024-04-06T02:48:10.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.359+0000 7fc74c679700 1 -- 172.21.15.105:0/2809768704 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fc74419aac0 con 0x7fc744100140 2024-04-06T02:48:10.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.359+0000 7fc73b7fe700 1 -- 172.21.15.105:0/2809768704 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fc73405a9b0 con 0x7fc744100140 2024-04-06T02:48:10.345 INFO:teuthology.orchestra.run.smithi105.stdout:171798691852 2024-04-06T02:48:10.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.361+0000 7fc74c679700 1 -- 172.21.15.105:0/2809768704 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fc73006c4e0 msgr2=0x7fc73006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:10.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.361+0000 7fc74c679700 1 --2- 172.21.15.105:0/2809768704 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fc73006c4e0 0x7fc73006e9a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc744197d00 tx=0x7fc740008040 comp rx=0 tx=0).stop 2024-04-06T02:48:10.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.361+0000 7fc74c679700 1 -- 172.21.15.105:0/2809768704 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc744100140 msgr2=0x7fc7441966e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:10.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.361+0000 7fc74c679700 1 --2- 172.21.15.105:0/2809768704 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc744100140 0x7fc7441966e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc734006010 tx=0x7fc73400be20 comp rx=0 tx=0).stop 2024-04-06T02:48:10.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.361+0000 7fc74c679700 1 -- 172.21.15.105:0/2809768704 shutdown_connections 2024-04-06T02:48:10.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.361+0000 7fc74c679700 1 --2- 172.21.15.105:0/2809768704 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fc73006c4e0 0x7fc73006e9a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:10.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.361+0000 7fc74c679700 1 --2- 172.21.15.105:0/2809768704 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc744100140 0x7fc7441966e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:10.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.361+0000 7fc74c679700 1 --2- 172.21.15.105:0/2809768704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc744104a50 0x7fc744196c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:10.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.361+0000 7fc74c679700 1 -- 172.21.15.105:0/2809768704 >> 172.21.15.105:0/2809768704 conn(0x7fc744075e30 msgr2=0x7fc7440fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:10.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.361+0000 7fc74c679700 1 -- 172.21.15.105:0/2809768704 shutdown_connections 2024-04-06T02:48:10.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.361+0000 7fc74c679700 1 -- 172.21.15.105:0/2809768704 wait complete. 2024-04-06T02:48:10.763 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:10 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/2809768704' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-06T02:48:10.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.803+0000 7f4488752700 1 -- 172.21.15.105:0/1002689334 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4480100dc0 msgr2=0x7f44801011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:10.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.803+0000 7f4488752700 1 --2- 172.21.15.105:0/1002689334 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4480100dc0 0x7f44801011a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f447800b0a0 tx=0x7f447800b3b0 comp rx=0 tx=0).stop 2024-04-06T02:48:10.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.803+0000 7f4488752700 1 -- 172.21.15.105:0/1002689334 shutdown_connections 2024-04-06T02:48:10.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.803+0000 7f4488752700 1 --2- 172.21.15.105:0/1002689334 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4480101770 0x7f44801057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:10.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.803+0000 7f4488752700 1 --2- 172.21.15.105:0/1002689334 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4480100dc0 0x7f44801011a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:10.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.803+0000 7f4488752700 1 -- 172.21.15.105:0/1002689334 >> 172.21.15.105:0/1002689334 conn(0x7f44800fc650 msgr2=0x7f44800fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:10.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.803+0000 7f4488752700 1 -- 172.21.15.105:0/1002689334 shutdown_connections 2024-04-06T02:48:10.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.803+0000 7f4488752700 1 -- 172.21.15.105:0/1002689334 wait complete. 2024-04-06T02:48:10.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.803+0000 7f4488752700 1 Processor -- start 2024-04-06T02:48:10.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.803+0000 7f4488752700 1 -- start start 2024-04-06T02:48:10.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.803+0000 7f4488752700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4480100dc0 0x7f448019c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:10.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.804+0000 7f4488752700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4480101770 0x7f448019cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:10.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.804+0000 7f4488752700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f448019d210 con 0x7f4480100dc0 2024-04-06T02:48:10.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.804+0000 7f4488752700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44801966c0 con 0x7f4480101770 2024-04-06T02:48:10.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.804+0000 7f44864ee700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4480100dc0 0x7f448019c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:10.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.804+0000 7f44864ee700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4480100dc0 0x7f448019c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:37436/0 (socket says 172.21.15.105:37436) 2024-04-06T02:48:10.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.804+0000 7f44864ee700 1 -- 172.21.15.105:0/2028327194 learned_addr learned my addr 172.21.15.105:0/2028327194 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:48:10.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.804+0000 7f4485ced700 1 --2- 172.21.15.105:0/2028327194 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4480101770 0x7f448019cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:10.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.804+0000 7f44864ee700 1 -- 172.21.15.105:0/2028327194 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4480101770 msgr2=0x7f448019cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:10.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.804+0000 7f44864ee700 1 --2- 172.21.15.105:0/2028327194 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4480101770 0x7f448019cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:10.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.804+0000 7f44864ee700 1 -- 172.21.15.105:0/2028327194 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4478009d00 con 0x7f4480100dc0 2024-04-06T02:48:10.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.804+0000 7f44864ee700 1 --2- 172.21.15.105:0/2028327194 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4480100dc0 0x7f448019c640 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4478000f80 tx=0x7f447800cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:10.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.804+0000 7f44777fe700 1 -- 172.21.15.105:0/2028327194 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f447800e040 con 0x7f4480100dc0 2024-04-06T02:48:10.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.804+0000 7f44777fe700 1 -- 172.21.15.105:0/2028327194 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4478003de0 con 0x7f4480100dc0 2024-04-06T02:48:10.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.804+0000 7f4488752700 1 -- 172.21.15.105:0/2028327194 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4480196940 con 0x7f4480100dc0 2024-04-06T02:48:10.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.804+0000 7f44777fe700 1 -- 172.21.15.105:0/2028327194 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f447801d9a0 con 0x7f4480100dc0 2024-04-06T02:48:10.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.804+0000 7f4488752700 1 -- 172.21.15.105:0/2028327194 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4480196e30 con 0x7f4480100dc0 2024-04-06T02:48:10.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.805+0000 7f4488752700 1 -- 172.21.15.105:0/2028327194 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4480109160 con 0x7f4480100dc0 2024-04-06T02:48:10.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.806+0000 7f44777fe700 1 -- 172.21.15.105:0/2028327194 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f4478004040 con 0x7f4480100dc0 2024-04-06T02:48:10.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.806+0000 7f44777fe700 1 --2- 172.21.15.105:0/2028327194 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f446c06c4e0 0x7f446c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:10.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.806+0000 7f4485ced700 1 --2- 172.21.15.105:0/2028327194 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f446c06c4e0 0x7f446c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:10.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.806+0000 7f44777fe700 1 -- 172.21.15.105:0/2028327194 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f447808cdf0 con 0x7f4480100dc0 2024-04-06T02:48:10.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.807+0000 7f4485ced700 1 --2- 172.21.15.105:0/2028327194 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f446c06c4e0 0x7f446c06e9a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4480197b30 tx=0x7f4470009190 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:10.796 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:10.811+0000 7f44777fe700 1 -- 172.21.15.105:0/2028327194 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f447805b420 con 0x7f4480100dc0 2024-04-06T02:48:10.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:10 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/2809768704' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-06T02:48:10.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:11.004+0000 7f4488752700 1 -- 172.21.15.105:0/2028327194 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f448004ea90 con 0x7f4480100dc0 2024-04-06T02:48:10.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:11.005+0000 7f44777fe700 1 -- 172.21.15.105:0/2028327194 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f447805afb0 con 0x7f4480100dc0 2024-04-06T02:48:10.990 INFO:teuthology.orchestra.run.smithi105.stdout:60129542176 2024-04-06T02:48:10.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:11.007+0000 7f4488752700 1 -- 172.21.15.105:0/2028327194 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f446c06c4e0 msgr2=0x7f446c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:10.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:11.007+0000 7f4488752700 1 --2- 172.21.15.105:0/2028327194 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f446c06c4e0 0x7f446c06e9a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4480197b30 tx=0x7f4470009190 comp rx=0 tx=0).stop 2024-04-06T02:48:10.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:11.007+0000 7f4488752700 1 -- 172.21.15.105:0/2028327194 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4480100dc0 msgr2=0x7f448019c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:10.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:11.007+0000 7f4488752700 1 --2- 172.21.15.105:0/2028327194 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4480100dc0 0x7f448019c640 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4478000f80 tx=0x7f447800cd00 comp rx=0 tx=0).stop 2024-04-06T02:48:10.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:11.007+0000 7f4488752700 1 -- 172.21.15.105:0/2028327194 shutdown_connections 2024-04-06T02:48:10.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:11.007+0000 7f4488752700 1 --2- 172.21.15.105:0/2028327194 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f446c06c4e0 0x7f446c06e9a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:10.993 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:11.007+0000 7f4488752700 1 --2- 172.21.15.105:0/2028327194 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4480100dc0 0x7f448019c640 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:10.993 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:11.007+0000 7f4488752700 1 --2- 172.21.15.105:0/2028327194 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4480101770 0x7f448019cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:10.993 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:11.007+0000 7f4488752700 1 -- 172.21.15.105:0/2028327194 >> 172.21.15.105:0/2028327194 conn(0x7f44800fc650 msgr2=0x7f44800fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:10.993 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:11.007+0000 7f4488752700 1 -- 172.21.15.105:0/2028327194 shutdown_connections 2024-04-06T02:48:10.993 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:11.007+0000 7f4488752700 1 -- 172.21.15.105:0/2028327194 wait complete. 2024-04-06T02:48:11.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:11 smithi105 ceph-mon[29228]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:11.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:11 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/2028327194' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-06T02:48:11.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:11 smithi184 ceph-mon[37439]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:11.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:11 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/2028327194' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-06T02:48:11.991 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691849 got 171798691852 for osd.5 2024-04-06T02:48:11.991 DEBUG:teuthology.parallel:result is None 2024-04-06T02:48:12.397 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542173 got 60129542176 for osd.1 2024-04-06T02:48:12.397 DEBUG:teuthology.parallel:result is None 2024-04-06T02:48:12.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.827+0000 7f7e30f67700 1 -- 172.21.15.105:0/3182984051 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e2c100140 msgr2=0x7f7e2c1005c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:12.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.827+0000 7f7e30f67700 1 --2- 172.21.15.105:0/3182984051 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e2c100140 0x7f7e2c1005c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7e14009910 tx=0x7f7e14009c20 comp rx=0 tx=0).stop 2024-04-06T02:48:12.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.827+0000 7f7e30f67700 1 -- 172.21.15.105:0/3182984051 shutdown_connections 2024-04-06T02:48:12.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.827+0000 7f7e30f67700 1 --2- 172.21.15.105:0/3182984051 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e2c100140 0x7f7e2c1005c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:12.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.827+0000 7f7e30f67700 1 --2- 172.21.15.105:0/3182984051 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e2c104a50 0x7f7e2c104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:12.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.827+0000 7f7e30f67700 1 -- 172.21.15.105:0/3182984051 >> 172.21.15.105:0/3182984051 conn(0x7f7e2c075e30 msgr2=0x7f7e2c0fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:12.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.827+0000 7f7e30f67700 1 -- 172.21.15.105:0/3182984051 shutdown_connections 2024-04-06T02:48:12.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.827+0000 7f7e30f67700 1 -- 172.21.15.105:0/3182984051 wait complete. 2024-04-06T02:48:12.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.828+0000 7f7e30f67700 1 Processor -- start 2024-04-06T02:48:12.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.828+0000 7f7e30f67700 1 -- start start 2024-04-06T02:48:12.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.828+0000 7f7e30f67700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e2c100140 0x7f7e2c196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:12.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.828+0000 7f7e30f67700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e2c104a50 0x7f7e2c196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:12.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.828+0000 7f7e30f67700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c197270 con 0x7f7e2c104a50 2024-04-06T02:48:12.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.828+0000 7f7e30f67700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c19b000 con 0x7f7e2c100140 2024-04-06T02:48:12.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.828+0000 7f7e21d9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e2c104a50 0x7f7e2c196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:12.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.828+0000 7f7e21d9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e2c104a50 0x7f7e2c196b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:37456/0 (socket says 172.21.15.105:37456) 2024-04-06T02:48:12.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.828+0000 7f7e21d9b700 1 -- 172.21.15.105:0/280657016 learned_addr learned my addr 172.21.15.105:0/280657016 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:48:12.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.828+0000 7f7e2a59c700 1 --2- 172.21.15.105:0/280657016 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e2c100140 0x7f7e2c196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:12.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.829+0000 7f7e21d9b700 1 -- 172.21.15.105:0/280657016 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e2c100140 msgr2=0x7f7e2c196650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:12.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.829+0000 7f7e21d9b700 1 --2- 172.21.15.105:0/280657016 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e2c100140 0x7f7e2c196650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:12.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.829+0000 7f7e21d9b700 1 -- 172.21.15.105:0/280657016 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e140095f0 con 0x7f7e2c104a50 2024-04-06T02:48:12.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.829+0000 7f7e21d9b700 1 --2- 172.21.15.105:0/280657016 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e2c104a50 0x7f7e2c196b90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7e14006010 tx=0x7f7e1400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:12.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.829+0000 7f7e23fff700 1 -- 172.21.15.105:0/280657016 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e1401d070 con 0x7f7e2c104a50 2024-04-06T02:48:12.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.829+0000 7f7e30f67700 1 -- 172.21.15.105:0/280657016 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e2c19b280 con 0x7f7e2c104a50 2024-04-06T02:48:12.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.829+0000 7f7e23fff700 1 -- 172.21.15.105:0/280657016 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7e1400f460 con 0x7f7e2c104a50 2024-04-06T02:48:12.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.829+0000 7f7e23fff700 1 -- 172.21.15.105:0/280657016 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e14017630 con 0x7f7e2c104a50 2024-04-06T02:48:12.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.829+0000 7f7e30f67700 1 -- 172.21.15.105:0/280657016 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e2c19b770 con 0x7f7e2c104a50 2024-04-06T02:48:12.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.830+0000 7f7e30f67700 1 -- 172.21.15.105:0/280657016 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e2c04ea90 con 0x7f7e2c104a50 2024-04-06T02:48:12.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.830+0000 7f7e23fff700 1 -- 172.21.15.105:0/280657016 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f7e1400f5d0 con 0x7f7e2c104a50 2024-04-06T02:48:12.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.831+0000 7f7e23fff700 1 --2- 172.21.15.105:0/280657016 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f7e1806c4e0 0x7f7e1806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:12.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.831+0000 7f7e23fff700 1 -- 172.21.15.105:0/280657016 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f7e1408c8d0 con 0x7f7e2c104a50 2024-04-06T02:48:12.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.831+0000 7f7e2a59c700 1 --2- 172.21.15.105:0/280657016 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f7e1806c4e0 0x7f7e1806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:12.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.831+0000 7f7e2a59c700 1 --2- 172.21.15.105:0/280657016 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f7e1806c4e0 0x7f7e1806e9a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c005950 tx=0x7f7e1c00b410 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:12.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:12.835+0000 7f7e23fff700 1 -- 172.21.15.105:0/280657016 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7e14057470 con 0x7f7e2c104a50 2024-04-06T02:48:13.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:13.025+0000 7f7e30f67700 1 -- 172.21.15.105:0/280657016 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f7e2c1979b0 con 0x7f7e2c104a50 2024-04-06T02:48:13.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:13.026+0000 7f7e23fff700 1 -- 172.21.15.105:0/280657016 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f7e1405aa90 con 0x7f7e2c104a50 2024-04-06T02:48:13.011 INFO:teuthology.orchestra.run.smithi105.stdout:115964117014 2024-04-06T02:48:13.014 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:13.028+0000 7f7e30f67700 1 -- 172.21.15.105:0/280657016 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f7e1806c4e0 msgr2=0x7f7e1806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:13.014 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:13.028+0000 7f7e30f67700 1 --2- 172.21.15.105:0/280657016 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f7e1806c4e0 0x7f7e1806e9a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c005950 tx=0x7f7e1c00b410 comp rx=0 tx=0).stop 2024-04-06T02:48:13.014 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:13.028+0000 7f7e30f67700 1 -- 172.21.15.105:0/280657016 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e2c104a50 msgr2=0x7f7e2c196b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:13.014 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:13.028+0000 7f7e30f67700 1 --2- 172.21.15.105:0/280657016 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e2c104a50 0x7f7e2c196b90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7e14006010 tx=0x7f7e1400bd70 comp rx=0 tx=0).stop 2024-04-06T02:48:13.014 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:13.028+0000 7f7e30f67700 1 -- 172.21.15.105:0/280657016 shutdown_connections 2024-04-06T02:48:13.014 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:13.028+0000 7f7e30f67700 1 --2- 172.21.15.105:0/280657016 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f7e1806c4e0 0x7f7e1806e9a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:13.014 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:13.028+0000 7f7e30f67700 1 --2- 172.21.15.105:0/280657016 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e2c100140 0x7f7e2c196650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:13.015 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:13.028+0000 7f7e30f67700 1 --2- 172.21.15.105:0/280657016 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e2c104a50 0x7f7e2c196b90 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:13.015 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:13.028+0000 7f7e30f67700 1 -- 172.21.15.105:0/280657016 >> 172.21.15.105:0/280657016 conn(0x7f7e2c075e30 msgr2=0x7f7e2c0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:13.015 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:13.028+0000 7f7e30f67700 1 -- 172.21.15.105:0/280657016 shutdown_connections 2024-04-06T02:48:13.015 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:13.028+0000 7f7e30f67700 1 -- 172.21.15.105:0/280657016 wait complete. 2024-04-06T02:48:13.596 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:13 smithi105 ceph-mon[29228]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:13.596 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:13 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/280657016' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-06T02:48:13.619 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117011 got 115964117014 for osd.3 2024-04-06T02:48:13.619 DEBUG:teuthology.parallel:result is None 2024-04-06T02:48:13.619 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-06T02:48:13.619 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-06T02:48:13.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:13 smithi184 ceph-mon[37439]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:13.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:13 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/280657016' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-06T02:48:13.829 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:48:15.544 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.558+0000 7fb7c4a42700 1 -- 172.21.15.105:0/4290286596 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7c00ffbd0 msgr2=0x7fb7c0100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:15.544 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.558+0000 7fb7c4a42700 1 --2- 172.21.15.105:0/4290286596 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7c00ffbd0 0x7fb7c0100050 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0009910 tx=0x7fb7b0009c20 comp rx=0 tx=0).stop 2024-04-06T02:48:15.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.558+0000 7fb7c4a42700 1 -- 172.21.15.105:0/4290286596 shutdown_connections 2024-04-06T02:48:15.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.558+0000 7fb7c4a42700 1 --2- 172.21.15.105:0/4290286596 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7c00ffbd0 0x7fb7c0100050 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:15.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.558+0000 7fb7c4a42700 1 --2- 172.21.15.105:0/4290286596 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7c0105a40 0x7fb7c0105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:15.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.558+0000 7fb7c4a42700 1 -- 172.21.15.105:0/4290286596 >> 172.21.15.105:0/4290286596 conn(0x7fb7c00755a0 msgr2=0x7fb7c00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:15.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.559+0000 7fb7c4a42700 1 -- 172.21.15.105:0/4290286596 shutdown_connections 2024-04-06T02:48:15.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.559+0000 7fb7c4a42700 1 -- 172.21.15.105:0/4290286596 wait complete. 2024-04-06T02:48:15.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.559+0000 7fb7c4a42700 1 Processor -- start 2024-04-06T02:48:15.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.560+0000 7fb7c4a42700 1 -- start start 2024-04-06T02:48:15.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.560+0000 7fb7c4a42700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7c00ffbd0 0x7fb7c0197530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:15.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.560+0000 7fb7c4a42700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7c0105a40 0x7fb7c0197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:15.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.560+0000 7fb7c4a42700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c0198150 con 0x7fb7c0105a40 2024-04-06T02:48:15.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.560+0000 7fb7c4a42700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c0198290 con 0x7fb7c00ffbd0 2024-04-06T02:48:15.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.560+0000 7fb7bdd9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7c0105a40 0x7fb7c0197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:15.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.560+0000 7fb7bdd9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7c0105a40 0x7fb7c0197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:37468/0 (socket says 172.21.15.105:37468) 2024-04-06T02:48:15.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.560+0000 7fb7bdd9b700 1 -- 172.21.15.105:0/1411307063 learned_addr learned my addr 172.21.15.105:0/1411307063 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:48:15.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.560+0000 7fb7be59c700 1 --2- 172.21.15.105:0/1411307063 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7c00ffbd0 0x7fb7c0197530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:15.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.560+0000 7fb7bdd9b700 1 -- 172.21.15.105:0/1411307063 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7c00ffbd0 msgr2=0x7fb7c0197530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:15.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.560+0000 7fb7bdd9b700 1 --2- 172.21.15.105:0/1411307063 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7c00ffbd0 0x7fb7c0197530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:15.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.560+0000 7fb7bdd9b700 1 -- 172.21.15.105:0/1411307063 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b00095f0 con 0x7fb7c0105a40 2024-04-06T02:48:15.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.561+0000 7fb7bdd9b700 1 --2- 172.21.15.105:0/1411307063 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7c0105a40 0x7fb7c0197a70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0006010 tx=0x7fb7b000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:15.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.561+0000 7fb7af7fe700 1 -- 172.21.15.105:0/1411307063 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7b001d070 con 0x7fb7c0105a40 2024-04-06T02:48:15.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.561+0000 7fb7af7fe700 1 -- 172.21.15.105:0/1411307063 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb7b000f460 con 0x7fb7c0105a40 2024-04-06T02:48:15.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.561+0000 7fb7c4a42700 1 -- 172.21.15.105:0/1411307063 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c01a05a0 con 0x7fb7c0105a40 2024-04-06T02:48:15.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.561+0000 7fb7af7fe700 1 -- 172.21.15.105:0/1411307063 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7b0017630 con 0x7fb7c0105a40 2024-04-06T02:48:15.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.561+0000 7fb7c4a42700 1 -- 172.21.15.105:0/1411307063 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c0195650 con 0x7fb7c0105a40 2024-04-06T02:48:15.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.562+0000 7fb7c4a42700 1 -- 172.21.15.105:0/1411307063 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c0107fd0 con 0x7fb7c0105a40 2024-04-06T02:48:15.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.563+0000 7fb7af7fe700 1 -- 172.21.15.105:0/1411307063 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fb7b000f5d0 con 0x7fb7c0105a40 2024-04-06T02:48:15.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.563+0000 7fb7af7fe700 1 --2- 172.21.15.105:0/1411307063 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fb7a806c4e0 0x7fb7a806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:15.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.563+0000 7fb7af7fe700 1 -- 172.21.15.105:0/1411307063 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb7b008c940 con 0x7fb7c0105a40 2024-04-06T02:48:15.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.563+0000 7fb7be59c700 1 --2- 172.21.15.105:0/1411307063 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fb7a806c4e0 0x7fb7a806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:15.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.564+0000 7fb7be59c700 1 --2- 172.21.15.105:0/1411307063 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fb7a806c4e0 0x7fb7a806e9a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4005950 tx=0x7fb7b400b410 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:15.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.567+0000 7fb7af7fe700 1 -- 172.21.15.105:0/1411307063 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb7b0057460 con 0x7fb7c0105a40 2024-04-06T02:48:15.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.761+0000 7fb7c4a42700 1 -- 172.21.15.105:0/1411307063 --> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb7c002cce0 con 0x7fb7a806c4e0 2024-04-06T02:48:15.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.762+0000 7fb7af7fe700 1 -- 172.21.15.105:0/1411307063 <== mgr.14221 v2:172.21.15.105:6800/2334550011 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28076 (secure 0 0 0) 0x7fb7c002cce0 con 0x7fb7a806c4e0 2024-04-06T02:48:15.748 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:48:15.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.764+0000 7fb7c4a42700 1 -- 172.21.15.105:0/1411307063 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fb7a806c4e0 msgr2=0x7fb7a806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:15.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.764+0000 7fb7c4a42700 1 --2- 172.21.15.105:0/1411307063 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fb7a806c4e0 0x7fb7a806e9a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4005950 tx=0x7fb7b400b410 comp rx=0 tx=0).stop 2024-04-06T02:48:15.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.764+0000 7fb7c4a42700 1 -- 172.21.15.105:0/1411307063 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7c0105a40 msgr2=0x7fb7c0197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:15.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.764+0000 7fb7c4a42700 1 --2- 172.21.15.105:0/1411307063 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7c0105a40 0x7fb7c0197a70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0006010 tx=0x7fb7b000bd70 comp rx=0 tx=0).stop 2024-04-06T02:48:15.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.764+0000 7fb7c4a42700 1 -- 172.21.15.105:0/1411307063 shutdown_connections 2024-04-06T02:48:15.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.764+0000 7fb7c4a42700 1 --2- 172.21.15.105:0/1411307063 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fb7a806c4e0 0x7fb7a806e9a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:15.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.764+0000 7fb7c4a42700 1 --2- 172.21.15.105:0/1411307063 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7c00ffbd0 0x7fb7c0197530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:15.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.764+0000 7fb7c4a42700 1 --2- 172.21.15.105:0/1411307063 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7c0105a40 0x7fb7c0197a70 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:15.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.764+0000 7fb7c4a42700 1 -- 172.21.15.105:0/1411307063 >> 172.21.15.105:0/1411307063 conn(0x7fb7c00755a0 msgr2=0x7fb7c00fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:15.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.764+0000 7fb7c4a42700 1 -- 172.21.15.105:0/1411307063 shutdown_connections 2024-04-06T02:48:15.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:15.764+0000 7fb7c4a42700 1 -- 172.21.15.105:0/1411307063 wait complete. 2024-04-06T02:48:15.751 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-06T02:48:15.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:15 smithi105 ceph-mon[29228]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:15.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:15 smithi184 ceph-mon[37439]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:16.351 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-04-06T02:48:14.622661+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165636,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305276,"statfs":{"total":575970213888,"available":575800602624,"internally_reserved":0,"allocated":4018176,"data_stored":2627442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002234"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":187,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-06T02:47:25.585252+0000","last_change":"2024-04-06T02:47:00.344019+0000","last_active":"2024-04-06T02:47:25.585252+0000","last_peered":"2024-04-06T02:47:25.585252+0000","last_clean":"2024-04-06T02:47:25.585252+0000","last_became_active":"2024-04-06T02:47:00.343642+0000","last_became_peered":"2024-04-06T02:47:00.343642+0000","last_unstale":"2024-04-06T02:47:25.585252+0000","last_undegraded":"2024-04-06T02:47:25.585252+0000","last_fullsized":"2024-04-06T02:47:25.585252+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T02:46:13.361043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T02:46:13.361043+0000","last_clean_scrub_stamp":"2024-04-06T02:46:13.361043+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:05:30.412652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]}]},{"osd":4,"up_from":35,"seq":150323855377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 02:48:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.439}]},{"osd":1,"last update":"Sat Apr 6 02:48:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.49399999999999999}]},{"osd":2,"last update":"Sat Apr 6 02:48:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.46300000000000002}]},{"osd":3,"last update":"Sat Apr 6 02:48:01 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.33700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]}]},{"osd":3,"up_from":27,"seq":115964117014,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 02:47:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.36199999999999999}]},{"osd":1,"last update":"Sat Apr 6 02:47:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.495}]},{"osd":2,"last update":"Sat Apr 6 02:47:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.44600000000000001}]},{"osd":4,"last update":"Sat Apr 6 02:48:02 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.53600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 02:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.51000000000000001}]},{"osd":1,"last update":"Sat Apr 6 02:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.52300000000000002}]},{"osd":3,"last update":"Sat Apr 6 02:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.86099999999999999}]},{"osd":4,"last update":"Sat Apr 6 02:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.92800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 02:47:46 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":0.48099999999999998,"5min":0.504,"15min":0.504},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.34599999999999997,"15min":0.34000000000000002},"min":{"1min":0.224,"5min":0.189,"15min":0.189},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.34300000000000003}]},{"osd":2,"last update":"Sat Apr 6 02:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.504,"5min":0.504,"15min":0.504},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.38800000000000001}]},{"osd":3,"last update":"Sat Apr 6 02:47:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.436}]},{"osd":4,"last update":"Sat Apr 6 02:48:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.42299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 02:47:46 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.32800000000000001,"15min":0.32000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.31},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.34899999999999998,"15min":0.34000000000000002},"min":{"1min":0.26100000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.53500000000000003,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.25700000000000001}]},{"osd":2,"last update":"Sat Apr 6 02:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.24299999999999999}]},{"osd":3,"last update":"Sat Apr 6 02:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.35199999999999998}]},{"osd":4,"last update":"Sat Apr 6 02:48:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.40899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T02:48:16.352 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-06T02:48:16.560 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:48:16.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:16 smithi105 ceph-mon[29228]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T02:48:16.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:16 smithi184 ceph-mon[37439]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T02:48:17.921 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:17 smithi105 ceph-mon[29228]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:17.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:17 smithi184 ceph-mon[37439]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:18.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.153+0000 7f45eb8d0700 1 -- 172.21.15.105:0/248046014 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f45e4106c10 msgr2=0x7f45e4106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:18.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.153+0000 7f45eb8d0700 1 --2- 172.21.15.105:0/248046014 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f45e4106c10 0x7f45e4106ff0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f45e0009910 tx=0x7f45e0009c20 comp rx=0 tx=0).stop 2024-04-06T02:48:18.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.153+0000 7f45eb8d0700 1 -- 172.21.15.105:0/248046014 shutdown_connections 2024-04-06T02:48:18.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.153+0000 7f45eb8d0700 1 --2- 172.21.15.105:0/248046014 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f45e40ff310 0x7f45e40ff770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:18.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.153+0000 7f45eb8d0700 1 --2- 172.21.15.105:0/248046014 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f45e4106c10 0x7f45e4106ff0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:18.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.153+0000 7f45eb8d0700 1 -- 172.21.15.105:0/248046014 >> 172.21.15.105:0/248046014 conn(0x7f45e4074bf0 msgr2=0x7f45e4075000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:18.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.153+0000 7f45eb8d0700 1 -- 172.21.15.105:0/248046014 shutdown_connections 2024-04-06T02:48:18.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.153+0000 7f45eb8d0700 1 -- 172.21.15.105:0/248046014 wait complete. 2024-04-06T02:48:18.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.153+0000 7f45eb8d0700 1 Processor -- start 2024-04-06T02:48:18.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.153+0000 7f45eb8d0700 1 -- start start 2024-04-06T02:48:18.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.154+0000 7f45eb8d0700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f45e40ff310 0x7f45e41966f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:18.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.154+0000 7f45eb8d0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f45e4106c10 0x7f45e4196c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:18.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.154+0000 7f45eb8d0700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e41972c0 con 0x7f45e4106c10 2024-04-06T02:48:18.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.154+0000 7f45eb8d0700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e4199ff0 con 0x7f45e40ff310 2024-04-06T02:48:18.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.154+0000 7f45e8e6b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f45e4106c10 0x7f45e4196c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:18.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.154+0000 7f45e8e6b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f45e4106c10 0x7f45e4196c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:37486/0 (socket says 172.21.15.105:37486) 2024-04-06T02:48:18.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.154+0000 7f45e8e6b700 1 -- 172.21.15.105:0/637673742 learned_addr learned my addr 172.21.15.105:0/637673742 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:48:18.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.154+0000 7f45e966c700 1 --2- 172.21.15.105:0/637673742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f45e40ff310 0x7f45e41966f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:18.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.154+0000 7f45e8e6b700 1 -- 172.21.15.105:0/637673742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f45e40ff310 msgr2=0x7f45e41966f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:18.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.154+0000 7f45e8e6b700 1 --2- 172.21.15.105:0/637673742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f45e40ff310 0x7f45e41966f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:18.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.154+0000 7f45e8e6b700 1 -- 172.21.15.105:0/637673742 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e00095f0 con 0x7f45e4106c10 2024-04-06T02:48:18.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.154+0000 7f45e966c700 1 --2- 172.21.15.105:0/637673742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f45e40ff310 0x7f45e41966f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T02:48:18.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.154+0000 7f45e8e6b700 1 --2- 172.21.15.105:0/637673742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f45e4106c10 0x7f45e4196c30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f45d400c8a0 tx=0x7f45d400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:18.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.154+0000 7f45da7fc700 1 -- 172.21.15.105:0/637673742 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45d400cea0 con 0x7f45e4106c10 2024-04-06T02:48:18.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.154+0000 7f45eb8d0700 1 -- 172.21.15.105:0/637673742 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45e419a2d0 con 0x7f45e4106c10 2024-04-06T02:48:18.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.154+0000 7f45da7fc700 1 -- 172.21.15.105:0/637673742 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f45d4004830 con 0x7f45e4106c10 2024-04-06T02:48:18.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.154+0000 7f45da7fc700 1 -- 172.21.15.105:0/637673742 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45d40055b0 con 0x7f45e4106c10 2024-04-06T02:48:18.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.154+0000 7f45eb8d0700 1 -- 172.21.15.105:0/637673742 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45e419a820 con 0x7f45e4106c10 2024-04-06T02:48:18.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.155+0000 7f45eb8d0700 1 -- 172.21.15.105:0/637673742 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45e4109110 con 0x7f45e4106c10 2024-04-06T02:48:18.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.156+0000 7f45da7fc700 1 -- 172.21.15.105:0/637673742 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f45d4007ad0 con 0x7f45e4106c10 2024-04-06T02:48:18.143 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.156+0000 7f45da7fc700 1 --2- 172.21.15.105:0/637673742 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f45d006c4e0 0x7f45d006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:18.143 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.156+0000 7f45e966c700 1 --2- 172.21.15.105:0/637673742 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f45d006c4e0 0x7f45d006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:18.143 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.156+0000 7f45da7fc700 1 -- 172.21.15.105:0/637673742 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f45d408b240 con 0x7f45e4106c10 2024-04-06T02:48:18.143 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.157+0000 7f45e966c700 1 --2- 172.21.15.105:0/637673742 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f45d006c4e0 0x7f45d006e9a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f45e000b5c0 tx=0x7f45e001a040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:18.146 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.161+0000 7f45da7fc700 1 -- 172.21.15.105:0/637673742 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f45d4055d60 con 0x7f45e4106c10 2024-04-06T02:48:18.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.346+0000 7f45eb8d0700 1 -- 172.21.15.105:0/637673742 --> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f45e419aaa0 con 0x7f45d006c4e0 2024-04-06T02:48:18.333 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.348+0000 7f45da7fc700 1 -- 172.21.15.105:0/637673742 <== mgr.14221 v2:172.21.15.105:6800/2334550011 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28048 (secure 0 0 0) 0x7f45e419aaa0 con 0x7f45d006c4e0 2024-04-06T02:48:18.333 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:48:18.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.350+0000 7f45eb8d0700 1 -- 172.21.15.105:0/637673742 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f45d006c4e0 msgr2=0x7f45d006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:18.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.350+0000 7f45eb8d0700 1 --2- 172.21.15.105:0/637673742 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f45d006c4e0 0x7f45d006e9a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f45e000b5c0 tx=0x7f45e001a040 comp rx=0 tx=0).stop 2024-04-06T02:48:18.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.350+0000 7f45eb8d0700 1 -- 172.21.15.105:0/637673742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f45e4106c10 msgr2=0x7f45e4196c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:18.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.350+0000 7f45eb8d0700 1 --2- 172.21.15.105:0/637673742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f45e4106c10 0x7f45e4196c30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f45d400c8a0 tx=0x7f45d400cc60 comp rx=0 tx=0).stop 2024-04-06T02:48:18.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.350+0000 7f45eb8d0700 1 -- 172.21.15.105:0/637673742 shutdown_connections 2024-04-06T02:48:18.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.350+0000 7f45eb8d0700 1 --2- 172.21.15.105:0/637673742 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f45d006c4e0 0x7f45d006e9a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:18.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.350+0000 7f45eb8d0700 1 --2- 172.21.15.105:0/637673742 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f45e40ff310 0x7f45e41966f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:18.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.350+0000 7f45eb8d0700 1 --2- 172.21.15.105:0/637673742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f45e4106c10 0x7f45e4196c30 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:18.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.350+0000 7f45eb8d0700 1 -- 172.21.15.105:0/637673742 >> 172.21.15.105:0/637673742 conn(0x7f45e4074bf0 msgr2=0x7f45e40fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:18.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.350+0000 7f45eb8d0700 1 -- 172.21.15.105:0/637673742 shutdown_connections 2024-04-06T02:48:18.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:18.350+0000 7f45eb8d0700 1 -- 172.21.15.105:0/637673742 wait complete. 2024-04-06T02:48:18.336 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-06T02:48:19.032 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-04-06T02:48:16.623118+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165636,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305276,"statfs":{"total":575970213888,"available":575800602624,"internally_reserved":0,"allocated":4018176,"data_stored":2627442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002338"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":187,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-06T02:47:25.585252+0000","last_change":"2024-04-06T02:47:00.344019+0000","last_active":"2024-04-06T02:47:25.585252+0000","last_peered":"2024-04-06T02:47:25.585252+0000","last_clean":"2024-04-06T02:47:25.585252+0000","last_became_active":"2024-04-06T02:47:00.343642+0000","last_became_peered":"2024-04-06T02:47:00.343642+0000","last_unstale":"2024-04-06T02:47:25.585252+0000","last_undegraded":"2024-04-06T02:47:25.585252+0000","last_fullsized":"2024-04-06T02:47:25.585252+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T02:46:13.361043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T02:46:13.361043+0000","last_clean_scrub_stamp":"2024-04-06T02:46:13.361043+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T04:05:30.412652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]}]},{"osd":4,"up_from":35,"seq":150323855378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 02:48:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.51300000000000001}]},{"osd":1,"last update":"Sat Apr 6 02:48:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.47599999999999998}]},{"osd":2,"last update":"Sat Apr 6 02:48:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.46200000000000002}]},{"osd":3,"last update":"Sat Apr 6 02:48:01 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.55000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]}]},{"osd":3,"up_from":27,"seq":115964117015,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 02:47:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.627}]},{"osd":1,"last update":"Sat Apr 6 02:47:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.496},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.56000000000000005}]},{"osd":2,"last update":"Sat Apr 6 02:47:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.47499999999999998}]},{"osd":4,"last update":"Sat Apr 6 02:48:02 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.59499999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 02:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.51000000000000001}]},{"osd":1,"last update":"Sat Apr 6 02:47:11 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.52300000000000002}]},{"osd":3,"last update":"Sat Apr 6 02:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.86099999999999999}]},{"osd":4,"last update":"Sat Apr 6 02:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.92800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 02:47:46 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":0.48099999999999998,"5min":0.504,"15min":0.504},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.378,"5min":0.34599999999999997,"15min":0.34000000000000002},"min":{"1min":0.224,"5min":0.189,"15min":0.189},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.317}]},{"osd":2,"last update":"Sat Apr 6 02:47:17 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.504,"5min":0.504,"15min":0.504},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.33600000000000002}]},{"osd":3,"last update":"Sat Apr 6 02:47:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.749}]},{"osd":4,"last update":"Sat Apr 6 02:48:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.52700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 02:47:46 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.32800000000000001,"15min":0.32000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.31},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.34899999999999998,"15min":0.34000000000000002},"min":{"1min":0.26100000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.53500000000000003,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.25700000000000001}]},{"osd":2,"last update":"Sat Apr 6 02:47:13 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.24299999999999999}]},{"osd":3,"last update":"Sat Apr 6 02:47:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.35199999999999998}]},{"osd":4,"last update":"Sat Apr 6 02:48:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.40899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T02:48:19.033 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-06T02:48:19.033 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-06T02:48:19.034 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-06T02:48:19.034 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph health --format=json 2024-04-06T02:48:19.239 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:48:19.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:19 smithi105 ceph-mon[29228]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T02:48:19.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:19 smithi105 ceph-mon[29228]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:19.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:19 smithi184 ceph-mon[37439]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T02:48:19.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:19 smithi184 ceph-mon[37439]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:20.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.833+0000 7f4c21934700 1 -- 172.21.15.105:0/3439128552 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c1c100620 msgr2=0x7f4c1c100a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:20.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.833+0000 7f4c21934700 1 --2- 172.21.15.105:0/3439128552 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c1c100620 0x7f4c1c100a00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c009910 tx=0x7f4c0c009c20 comp rx=0 tx=0).stop 2024-04-06T02:48:20.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.833+0000 7f4c21934700 1 -- 172.21.15.105:0/3439128552 shutdown_connections 2024-04-06T02:48:20.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.833+0000 7f4c21934700 1 --2- 172.21.15.105:0/3439128552 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c1c100fd0 0x7f4c1c1046e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:20.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.833+0000 7f4c21934700 1 --2- 172.21.15.105:0/3439128552 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c1c100620 0x7f4c1c100a00 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:20.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.833+0000 7f4c21934700 1 -- 172.21.15.105:0/3439128552 >> 172.21.15.105:0/3439128552 conn(0x7f4c1c074d80 msgr2=0x7f4c1c075190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:20.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.833+0000 7f4c21934700 1 -- 172.21.15.105:0/3439128552 shutdown_connections 2024-04-06T02:48:20.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.833+0000 7f4c21934700 1 -- 172.21.15.105:0/3439128552 wait complete. 2024-04-06T02:48:20.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.833+0000 7f4c21934700 1 Processor -- start 2024-04-06T02:48:20.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.833+0000 7f4c21934700 1 -- start start 2024-04-06T02:48:20.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.834+0000 7f4c21934700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c1c100620 0x7f4c1c195410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:20.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.834+0000 7f4c21934700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c1c100fd0 0x7f4c1c195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:20.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.834+0000 7f4c21934700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1c195e90 con 0x7f4c1c100fd0 2024-04-06T02:48:20.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.834+0000 7f4c21934700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1c196000 con 0x7f4c1c100620 2024-04-06T02:48:20.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.834+0000 7f4c127fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c1c100fd0 0x7f4c1c195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:20.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.834+0000 7f4c127fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c1c100fd0 0x7f4c1c195950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57338/0 (socket says 172.21.15.105:57338) 2024-04-06T02:48:20.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.834+0000 7f4c127fc700 1 -- 172.21.15.105:0/2871791482 learned_addr learned my addr 172.21.15.105:0/2871791482 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:48:20.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.834+0000 7f4c1affd700 1 --2- 172.21.15.105:0/2871791482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c1c100620 0x7f4c1c195410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:20.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.834+0000 7f4c127fc700 1 -- 172.21.15.105:0/2871791482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c1c100620 msgr2=0x7f4c1c195410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:20.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.834+0000 7f4c127fc700 1 --2- 172.21.15.105:0/2871791482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c1c100620 0x7f4c1c195410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:20.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.834+0000 7f4c127fc700 1 -- 172.21.15.105:0/2871791482 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0c0095f0 con 0x7f4c1c100fd0 2024-04-06T02:48:20.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.834+0000 7f4c127fc700 1 --2- 172.21.15.105:0/2871791482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c1c100fd0 0x7f4c1c195950 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4c0400d900 tx=0x7f4c0400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:20.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.834+0000 7f4c20932700 1 -- 172.21.15.105:0/2871791482 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c04009900 con 0x7f4c1c100fd0 2024-04-06T02:48:20.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.834+0000 7f4c20932700 1 -- 172.21.15.105:0/2871791482 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4c04010460 con 0x7f4c1c100fd0 2024-04-06T02:48:20.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.834+0000 7f4c21934700 1 -- 172.21.15.105:0/2871791482 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c1c1a2a80 con 0x7f4c1c100fd0 2024-04-06T02:48:20.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.834+0000 7f4c20932700 1 -- 172.21.15.105:0/2871791482 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c0400f5f0 con 0x7f4c1c100fd0 2024-04-06T02:48:20.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.834+0000 7f4c21934700 1 -- 172.21.15.105:0/2871791482 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c1c1a2de0 con 0x7f4c1c100fd0 2024-04-06T02:48:20.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.835+0000 7f4c21934700 1 -- 172.21.15.105:0/2871791482 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c1c108050 con 0x7f4c1c100fd0 2024-04-06T02:48:20.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.836+0000 7f4c20932700 1 -- 172.21.15.105:0/2871791482 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f4c040105d0 con 0x7f4c1c100fd0 2024-04-06T02:48:20.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.836+0000 7f4c20932700 1 --2- 172.21.15.105:0/2871791482 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f4c0806c4e0 0x7f4c0806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:20.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.836+0000 7f4c20932700 1 -- 172.21.15.105:0/2871791482 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f4c0408af00 con 0x7f4c1c100fd0 2024-04-06T02:48:20.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.836+0000 7f4c1affd700 1 --2- 172.21.15.105:0/2871791482 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f4c0806c4e0 0x7f4c0806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:20.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.836+0000 7f4c1affd700 1 --2- 172.21.15.105:0/2871791482 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f4c0806c4e0 0x7f4c0806e9a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c006010 tx=0x7f4c0c01a040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:20.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:20.841+0000 7f4c20932700 1 -- 172.21.15.105:0/2871791482 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4c04059530 con 0x7f4c1c100fd0 2024-04-06T02:48:21.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:21.071+0000 7f4c21934700 1 -- 172.21.15.105:0/2871791482 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4c1c04ea90 con 0x7f4c1c100fd0 2024-04-06T02:48:21.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:21.072+0000 7f4c20932700 1 -- 172.21.15.105:0/2871791482 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f4c04020040 con 0x7f4c1c100fd0 2024-04-06T02:48:21.057 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:48:21.057 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-06T02:48:21.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:21.074+0000 7f4c21934700 1 -- 172.21.15.105:0/2871791482 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f4c0806c4e0 msgr2=0x7f4c0806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:21.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:21.074+0000 7f4c21934700 1 --2- 172.21.15.105:0/2871791482 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f4c0806c4e0 0x7f4c0806e9a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c006010 tx=0x7f4c0c01a040 comp rx=0 tx=0).stop 2024-04-06T02:48:21.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:21.074+0000 7f4c21934700 1 -- 172.21.15.105:0/2871791482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c1c100fd0 msgr2=0x7f4c1c195950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:21.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:21.074+0000 7f4c21934700 1 --2- 172.21.15.105:0/2871791482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c1c100fd0 0x7f4c1c195950 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4c0400d900 tx=0x7f4c0400dc10 comp rx=0 tx=0).stop 2024-04-06T02:48:21.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:21.074+0000 7f4c21934700 1 -- 172.21.15.105:0/2871791482 shutdown_connections 2024-04-06T02:48:21.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:21.074+0000 7f4c21934700 1 --2- 172.21.15.105:0/2871791482 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f4c0806c4e0 0x7f4c0806e9a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:21.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:21.074+0000 7f4c21934700 1 --2- 172.21.15.105:0/2871791482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c1c100620 0x7f4c1c195410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:21.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:21.074+0000 7f4c21934700 1 --2- 172.21.15.105:0/2871791482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c1c100fd0 0x7f4c1c195950 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:21.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:21.074+0000 7f4c21934700 1 -- 172.21.15.105:0/2871791482 >> 172.21.15.105:0/2871791482 conn(0x7f4c1c074d80 msgr2=0x7f4c1c0fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:21.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:21.074+0000 7f4c21934700 1 -- 172.21.15.105:0/2871791482 shutdown_connections 2024-04-06T02:48:21.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:21.074+0000 7f4c21934700 1 -- 172.21.15.105:0/2871791482 wait complete. 2024-04-06T02:48:21.740 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-06T02:48:21.740 INFO:tasks.cephadm:Setup complete, yielding 2024-04-06T02:48:21.740 INFO:teuthology.run_tasks:Running task print... 2024-04-06T02:48:21.750 INFO:teuthology.task.print:**** done end installing v18.2.1 cephadm ... 2024-04-06T02:48:21.750 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-06T02:48:21.796 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi105.front.sepia.ceph.com 2024-04-06T02:48:21.797 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-04-06T02:48:21.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:21 smithi184 ceph-mon[37439]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:21.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:21 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/2871791482' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-06T02:48:22.001 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:21 smithi105 ceph-mon[29228]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:22.001 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:21 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/2871791482' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-06T02:48:22.007 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:48:23.608 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.621+0000 7f8e34f39700 1 -- 172.21.15.105:0/3799723383 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8e30100c00 msgr2=0x7f8e30101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:23.608 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.621+0000 7f8e34f39700 1 --2- 172.21.15.105:0/3799723383 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8e30100c00 0x7f8e30101020 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8e20009910 tx=0x7f8e20009c20 comp rx=0 tx=0).stop 2024-04-06T02:48:23.608 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.621+0000 7f8e34f39700 1 -- 172.21.15.105:0/3799723383 shutdown_connections 2024-04-06T02:48:23.608 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.621+0000 7f8e34f39700 1 --2- 172.21.15.105:0/3799723383 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e30101e00 0x7f8e30102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:23.608 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.621+0000 7f8e34f39700 1 --2- 172.21.15.105:0/3799723383 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8e30100c00 0x7f8e30101020 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:23.609 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.621+0000 7f8e34f39700 1 -- 172.21.15.105:0/3799723383 >> 172.21.15.105:0/3799723383 conn(0x7f8e300fc180 msgr2=0x7f8e300fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:23.609 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.621+0000 7f8e34f39700 1 -- 172.21.15.105:0/3799723383 shutdown_connections 2024-04-06T02:48:23.609 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.621+0000 7f8e34f39700 1 -- 172.21.15.105:0/3799723383 wait complete. 2024-04-06T02:48:23.609 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.622+0000 7f8e34f39700 1 Processor -- start 2024-04-06T02:48:23.609 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.622+0000 7f8e34f39700 1 -- start start 2024-04-06T02:48:23.609 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.622+0000 7f8e34f39700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e30100c00 0x7f8e30196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:23.609 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.622+0000 7f8e34f39700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8e30101e00 0x7f8e301968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:23.609 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.622+0000 7f8e34f39700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e30196ef0 con 0x7f8e30101e00 2024-04-06T02:48:23.609 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.622+0000 7f8e34f39700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e30197030 con 0x7f8e30100c00 2024-04-06T02:48:23.609 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.622+0000 7f8e2dd9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8e30101e00 0x7f8e301968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:23.609 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.622+0000 7f8e2dd9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8e30101e00 0x7f8e301968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57344/0 (socket says 172.21.15.105:57344) 2024-04-06T02:48:23.609 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.622+0000 7f8e2dd9b700 1 -- 172.21.15.105:0/4076110689 learned_addr learned my addr 172.21.15.105:0/4076110689 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:48:23.610 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.622+0000 7f8e2e59c700 1 --2- 172.21.15.105:0/4076110689 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e30100c00 0x7f8e30196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:23.610 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.622+0000 7f8e2dd9b700 1 -- 172.21.15.105:0/4076110689 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e30100c00 msgr2=0x7f8e30196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:23.610 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.622+0000 7f8e2dd9b700 1 --2- 172.21.15.105:0/4076110689 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e30100c00 0x7f8e30196390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:23.610 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.622+0000 7f8e2dd9b700 1 -- 172.21.15.105:0/4076110689 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e200095f0 con 0x7f8e30101e00 2024-04-06T02:48:23.610 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.623+0000 7f8e2dd9b700 1 --2- 172.21.15.105:0/4076110689 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8e30101e00 0x7f8e301968d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8e2400c8a0 tx=0x7f8e2400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:23.610 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.623+0000 7f8e1f7fe700 1 -- 172.21.15.105:0/4076110689 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e240078e0 con 0x7f8e30101e00 2024-04-06T02:48:23.610 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.623+0000 7f8e34f39700 1 -- 172.21.15.105:0/4076110689 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e3019bae0 con 0x7f8e30101e00 2024-04-06T02:48:23.610 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.623+0000 7f8e1f7fe700 1 -- 172.21.15.105:0/4076110689 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8e2400f450 con 0x7f8e30101e00 2024-04-06T02:48:23.610 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.623+0000 7f8e1f7fe700 1 -- 172.21.15.105:0/4076110689 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e2400e5e0 con 0x7f8e30101e00 2024-04-06T02:48:23.610 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.623+0000 7f8e34f39700 1 -- 172.21.15.105:0/4076110689 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e30072fc0 con 0x7f8e30101e00 2024-04-06T02:48:23.611 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.624+0000 7f8e34f39700 1 -- 172.21.15.105:0/4076110689 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e30109760 con 0x7f8e30101e00 2024-04-06T02:48:23.611 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.625+0000 7f8e1f7fe700 1 -- 172.21.15.105:0/4076110689 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f8e2400e740 con 0x7f8e30101e00 2024-04-06T02:48:23.611 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.625+0000 7f8e1f7fe700 1 --2- 172.21.15.105:0/4076110689 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f8e180708f0 0x7f8e18072db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:23.611 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.625+0000 7f8e1f7fe700 1 -- 172.21.15.105:0/4076110689 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f8e2408b4e0 con 0x7f8e30101e00 2024-04-06T02:48:23.611 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.625+0000 7f8e2e59c700 1 --2- 172.21.15.105:0/4076110689 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f8e180708f0 0x7f8e18072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:23.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.626+0000 7f8e2e59c700 1 --2- 172.21.15.105:0/4076110689 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f8e180708f0 0x7f8e18072db0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8e20006010 tx=0x7f8e2000b540 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:23.615 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.630+0000 7f8e1f7fe700 1 -- 172.21.15.105:0/4076110689 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8e24059a90 con 0x7f8e30101e00 2024-04-06T02:48:23.638 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:23 smithi105 ceph-mon[29228]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:23.730 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:23 smithi184 ceph-mon[37439]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:23.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.830+0000 7f8e34f39700 1 -- 172.21.15.105:0/4076110689 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f8e30109950 con 0x7f8e30101e00 2024-04-06T02:48:23.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.834+0000 7f8e1f7fe700 1 -- 172.21.15.105:0/4076110689 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f8e24059620 con 0x7f8e30101e00 2024-04-06T02:48:23.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.837+0000 7f8e34f39700 1 -- 172.21.15.105:0/4076110689 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f8e180708f0 msgr2=0x7f8e18072db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:23.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.837+0000 7f8e34f39700 1 --2- 172.21.15.105:0/4076110689 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f8e180708f0 0x7f8e18072db0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8e20006010 tx=0x7f8e2000b540 comp rx=0 tx=0).stop 2024-04-06T02:48:23.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.837+0000 7f8e34f39700 1 -- 172.21.15.105:0/4076110689 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8e30101e00 msgr2=0x7f8e301968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:23.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.837+0000 7f8e34f39700 1 --2- 172.21.15.105:0/4076110689 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8e30101e00 0x7f8e301968d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8e2400c8a0 tx=0x7f8e2400cbb0 comp rx=0 tx=0).stop 2024-04-06T02:48:23.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.837+0000 7f8e34f39700 1 -- 172.21.15.105:0/4076110689 shutdown_connections 2024-04-06T02:48:23.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.837+0000 7f8e34f39700 1 --2- 172.21.15.105:0/4076110689 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f8e180708f0 0x7f8e18072db0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:23.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.837+0000 7f8e34f39700 1 --2- 172.21.15.105:0/4076110689 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e30100c00 0x7f8e30196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:23.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.837+0000 7f8e34f39700 1 --2- 172.21.15.105:0/4076110689 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8e30101e00 0x7f8e301968d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:23.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.837+0000 7f8e34f39700 1 -- 172.21.15.105:0/4076110689 >> 172.21.15.105:0/4076110689 conn(0x7f8e300fc180 msgr2=0x7f8e30105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:23.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.837+0000 7f8e34f39700 1 -- 172.21.15.105:0/4076110689 shutdown_connections 2024-04-06T02:48:23.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:23.837+0000 7f8e34f39700 1 -- 172.21.15.105:0/4076110689 wait complete. 2024-04-06T02:48:24.592 INFO:teuthology.run_tasks:Running task print... 2024-04-06T02:48:24.601 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-04-06T02:48:24.601 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-06T02:48:24.643 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi105.front.sepia.ceph.com 2024-04-06T02:48:24.643 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch status' 2024-04-06T02:48:24.852 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:48:24.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:24 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/4076110689' entity='client.admin' 2024-04-06T02:48:24.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:24 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:48:24.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:24 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:48:24.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:24 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:48:24.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:24 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:24.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:24 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:48:25.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:24 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/4076110689' entity='client.admin' 2024-04-06T02:48:25.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:24 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:48:25.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:24 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:48:25.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:24 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:48:25.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:24 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:25.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:24 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:48:25.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:25 smithi105 ceph-mon[29228]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:26.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:25 smithi184 ceph-mon[37439]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:26.533 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.545+0000 7f8663580700 1 -- 172.21.15.105:0/2431646773 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f865c103fe0 msgr2=0x7f865c1063d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:26.533 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.545+0000 7f8663580700 1 --2- 172.21.15.105:0/2431646773 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f865c103fe0 0x7f865c1063d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8650009910 tx=0x7f8650009c20 comp rx=0 tx=0).stop 2024-04-06T02:48:26.533 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.545+0000 7f8663580700 1 -- 172.21.15.105:0/2431646773 shutdown_connections 2024-04-06T02:48:26.533 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.545+0000 7f8663580700 1 --2- 172.21.15.105:0/2431646773 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f865c103fe0 0x7f865c1063d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:26.533 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.545+0000 7f8663580700 1 --2- 172.21.15.105:0/2431646773 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f865c1016b0 0x7f865c103aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:26.533 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.545+0000 7f8663580700 1 -- 172.21.15.105:0/2431646773 >> 172.21.15.105:0/2431646773 conn(0x7f865c0fb060 msgr2=0x7f865c0fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:26.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.545+0000 7f8663580700 1 -- 172.21.15.105:0/2431646773 shutdown_connections 2024-04-06T02:48:26.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.545+0000 7f8663580700 1 -- 172.21.15.105:0/2431646773 wait complete. 2024-04-06T02:48:26.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.545+0000 7f8663580700 1 Processor -- start 2024-04-06T02:48:26.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.546+0000 7f8663580700 1 -- start start 2024-04-06T02:48:26.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.546+0000 7f8663580700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f865c1016b0 0x7f865c196350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:26.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.546+0000 7f8663580700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f865c103fe0 0x7f865c196890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:26.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.546+0000 7f8663580700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865c196e20 con 0x7f865c103fe0 2024-04-06T02:48:26.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.546+0000 7f8663580700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865c196f60 con 0x7f865c1016b0 2024-04-06T02:48:26.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.546+0000 7f8660b1b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f865c103fe0 0x7f865c196890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:26.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.546+0000 7f8660b1b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f865c103fe0 0x7f865c196890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57352/0 (socket says 172.21.15.105:57352) 2024-04-06T02:48:26.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.546+0000 7f8660b1b700 1 -- 172.21.15.105:0/237000289 learned_addr learned my addr 172.21.15.105:0/237000289 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:48:26.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.546+0000 7f866131c700 1 --2- 172.21.15.105:0/237000289 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f865c1016b0 0x7f865c196350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:26.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.546+0000 7f8660b1b700 1 -- 172.21.15.105:0/237000289 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f865c1016b0 msgr2=0x7f865c196350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:26.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.546+0000 7f8660b1b700 1 --2- 172.21.15.105:0/237000289 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f865c1016b0 0x7f865c196350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:26.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.546+0000 7f8660b1b700 1 -- 172.21.15.105:0/237000289 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86500095f0 con 0x7f865c103fe0 2024-04-06T02:48:26.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.546+0000 7f866131c700 1 --2- 172.21.15.105:0/237000289 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f865c1016b0 0x7f865c196350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T02:48:26.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.546+0000 7f8660b1b700 1 --2- 172.21.15.105:0/237000289 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f865c103fe0 0x7f865c196890 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f865000bd70 tx=0x7f865000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:26.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.546+0000 7f865a7fc700 1 -- 172.21.15.105:0/237000289 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f865001d070 con 0x7f865c103fe0 2024-04-06T02:48:26.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.546+0000 7f8663580700 1 -- 172.21.15.105:0/237000289 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f865c19b9c0 con 0x7f865c103fe0 2024-04-06T02:48:26.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.546+0000 7f865a7fc700 1 -- 172.21.15.105:0/237000289 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8650022470 con 0x7f865c103fe0 2024-04-06T02:48:26.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.546+0000 7f865a7fc700 1 -- 172.21.15.105:0/237000289 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f865000f670 con 0x7f865c103fe0 2024-04-06T02:48:26.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.547+0000 7f8663580700 1 -- 172.21.15.105:0/237000289 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f865c19be80 con 0x7f865c103fe0 2024-04-06T02:48:26.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.547+0000 7f8663580700 1 -- 172.21.15.105:0/237000289 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f865c190530 con 0x7f865c103fe0 2024-04-06T02:48:26.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.548+0000 7f865a7fc700 1 -- 172.21.15.105:0/237000289 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f8650022a50 con 0x7f865c103fe0 2024-04-06T02:48:26.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.548+0000 7f865a7fc700 1 --2- 172.21.15.105:0/237000289 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f864406c4e0 0x7f864406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:26.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.548+0000 7f866131c700 1 --2- 172.21.15.105:0/237000289 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f864406c4e0 0x7f864406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:26.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.548+0000 7f865a7fc700 1 -- 172.21.15.105:0/237000289 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f865008cb90 con 0x7f865c103fe0 2024-04-06T02:48:26.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.549+0000 7f866131c700 1 --2- 172.21.15.105:0/237000289 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f864406c4e0 0x7f864406e9a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f864c00e300 tx=0x7f864c009040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:26.539 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.554+0000 7f865a7fc700 1 -- 172.21.15.105:0/237000289 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f865005b1f0 con 0x7f865c103fe0 2024-04-06T02:48:26.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.763+0000 7f8663580700 1 -- 172.21.15.105:0/237000289 --> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f865c0611d0 con 0x7f864406c4e0 2024-04-06T02:48:26.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.764+0000 7f865a7fc700 1 -- 172.21.15.105:0/237000289 <== mgr.14221 v2:172.21.15.105:6800/2334550011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f865c0611d0 con 0x7f864406c4e0 2024-04-06T02:48:26.750 INFO:teuthology.orchestra.run.smithi105.stdout:Backend: cephadm 2024-04-06T02:48:26.750 INFO:teuthology.orchestra.run.smithi105.stdout:Available: Yes 2024-04-06T02:48:26.750 INFO:teuthology.orchestra.run.smithi105.stdout:Paused: No 2024-04-06T02:48:26.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.766+0000 7f8663580700 1 -- 172.21.15.105:0/237000289 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f864406c4e0 msgr2=0x7f864406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:26.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.766+0000 7f8663580700 1 --2- 172.21.15.105:0/237000289 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f864406c4e0 0x7f864406e9a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f864c00e300 tx=0x7f864c009040 comp rx=0 tx=0).stop 2024-04-06T02:48:26.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.766+0000 7f8663580700 1 -- 172.21.15.105:0/237000289 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f865c103fe0 msgr2=0x7f865c196890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:26.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.766+0000 7f8663580700 1 --2- 172.21.15.105:0/237000289 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f865c103fe0 0x7f865c196890 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f865000bd70 tx=0x7f865000be50 comp rx=0 tx=0).stop 2024-04-06T02:48:26.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.766+0000 7f8663580700 1 -- 172.21.15.105:0/237000289 shutdown_connections 2024-04-06T02:48:26.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.766+0000 7f8663580700 1 --2- 172.21.15.105:0/237000289 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f864406c4e0 0x7f864406e9a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:26.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.766+0000 7f8663580700 1 --2- 172.21.15.105:0/237000289 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f865c1016b0 0x7f865c196350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:26.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.766+0000 7f8663580700 1 --2- 172.21.15.105:0/237000289 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f865c103fe0 0x7f865c196890 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:26.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.766+0000 7f8663580700 1 -- 172.21.15.105:0/237000289 >> 172.21.15.105:0/237000289 conn(0x7f865c0fb060 msgr2=0x7f865c0fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:26.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.767+0000 7f8663580700 1 -- 172.21.15.105:0/237000289 shutdown_connections 2024-04-06T02:48:26.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:26.767+0000 7f8663580700 1 -- 172.21.15.105:0/237000289 wait complete. 2024-04-06T02:48:27.484 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch ps' 2024-04-06T02:48:27.689 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:48:28.062 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:27 smithi105 ceph-mon[29228]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:28.062 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:27 smithi105 ceph-mon[29228]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:48:28.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:27 smithi184 ceph-mon[37439]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:28.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:27 smithi184 ceph-mon[37439]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:48:29.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.319+0000 7f4adf55e700 1 -- 172.21.15.105:0/1664292514 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ad80ffab0 msgr2=0x7f4ad80ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:29.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.319+0000 7f4adf55e700 1 --2- 172.21.15.105:0/1664292514 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ad80ffab0 0x7f4ad80ffed0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8009910 tx=0x7f4ac8009c20 comp rx=0 tx=0).stop 2024-04-06T02:48:29.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.319+0000 7f4adf55e700 1 -- 172.21.15.105:0/1664292514 shutdown_connections 2024-04-06T02:48:29.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.319+0000 7f4adf55e700 1 --2- 172.21.15.105:0/1664292514 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ad8100cb0 0x7f4ad8101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:29.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.319+0000 7f4adf55e700 1 --2- 172.21.15.105:0/1664292514 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ad80ffab0 0x7f4ad80ffed0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:29.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.319+0000 7f4adf55e700 1 -- 172.21.15.105:0/1664292514 >> 172.21.15.105:0/1664292514 conn(0x7f4ad80fb030 msgr2=0x7f4ad80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:29.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.319+0000 7f4adf55e700 1 -- 172.21.15.105:0/1664292514 shutdown_connections 2024-04-06T02:48:29.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.319+0000 7f4adf55e700 1 -- 172.21.15.105:0/1664292514 wait complete. 2024-04-06T02:48:29.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.320+0000 7f4adf55e700 1 Processor -- start 2024-04-06T02:48:29.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.320+0000 7f4adf55e700 1 -- start start 2024-04-06T02:48:29.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.320+0000 7f4adf55e700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ad80ffab0 0x7f4ad81963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:29.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.320+0000 7f4adf55e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ad8100cb0 0x7f4ad8196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:29.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.320+0000 7f4adf55e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad8196f50 con 0x7f4ad8100cb0 2024-04-06T02:48:29.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.320+0000 7f4adf55e700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad8197090 con 0x7f4ad80ffab0 2024-04-06T02:48:29.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.320+0000 7f4adcaf9700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ad8100cb0 0x7f4ad8196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:29.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.320+0000 7f4adcaf9700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ad8100cb0 0x7f4ad8196930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57362/0 (socket says 172.21.15.105:57362) 2024-04-06T02:48:29.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.320+0000 7f4adcaf9700 1 -- 172.21.15.105:0/1292487942 learned_addr learned my addr 172.21.15.105:0/1292487942 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:48:29.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.320+0000 7f4add2fa700 1 --2- 172.21.15.105:0/1292487942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ad80ffab0 0x7f4ad81963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:29.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.320+0000 7f4adcaf9700 1 -- 172.21.15.105:0/1292487942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ad80ffab0 msgr2=0x7f4ad81963f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:29.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.320+0000 7f4adcaf9700 1 --2- 172.21.15.105:0/1292487942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ad80ffab0 0x7f4ad81963f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:29.308 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.320+0000 7f4adcaf9700 1 -- 172.21.15.105:0/1292487942 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac80095f0 con 0x7f4ad8100cb0 2024-04-06T02:48:29.308 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.320+0000 7f4adcaf9700 1 --2- 172.21.15.105:0/1292487942 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ad8100cb0 0x7f4ad8196930 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4ad400b4b0 tx=0x7f4ad400b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:29.308 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.320+0000 7f4ace7fc700 1 -- 172.21.15.105:0/1292487942 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ad4011800 con 0x7f4ad8100cb0 2024-04-06T02:48:29.308 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.321+0000 7f4adf55e700 1 -- 172.21.15.105:0/1292487942 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ad819bb40 con 0x7f4ad8100cb0 2024-04-06T02:48:29.308 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.321+0000 7f4ace7fc700 1 -- 172.21.15.105:0/1292487942 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4ad4011e10 con 0x7f4ad8100cb0 2024-04-06T02:48:29.308 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.321+0000 7f4ace7fc700 1 -- 172.21.15.105:0/1292487942 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ad400f2e0 con 0x7f4ad8100cb0 2024-04-06T02:48:29.308 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.321+0000 7f4adf55e700 1 -- 172.21.15.105:0/1292487942 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ad819c110 con 0x7f4ad8100cb0 2024-04-06T02:48:29.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.322+0000 7f4adf55e700 1 -- 172.21.15.105:0/1292487942 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ad8108690 con 0x7f4ad8100cb0 2024-04-06T02:48:29.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.322+0000 7f4ace7fc700 1 -- 172.21.15.105:0/1292487942 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f4ad4010380 con 0x7f4ad8100cb0 2024-04-06T02:48:29.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.322+0000 7f4ace7fc700 1 --2- 172.21.15.105:0/1292487942 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f4ac406c4e0 0x7f4ac406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:29.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.322+0000 7f4add2fa700 1 --2- 172.21.15.105:0/1292487942 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f4ac406c4e0 0x7f4ac406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:29.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.322+0000 7f4ace7fc700 1 -- 172.21.15.105:0/1292487942 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f4ad408b090 con 0x7f4ad8100cb0 2024-04-06T02:48:29.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.323+0000 7f4add2fa700 1 --2- 172.21.15.105:0/1292487942 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f4ac406c4e0 0x7f4ac406e9a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8006010 tx=0x7f4ac800b540 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:29.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.327+0000 7f4ace7fc700 1 -- 172.21.15.105:0/1292487942 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4ad4059640 con 0x7f4ad8100cb0 2024-04-06T02:48:29.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.524+0000 7f4adf55e700 1 -- 172.21.15.105:0/1292487942 --> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ad8068b20 con 0x7f4ac406c4e0 2024-04-06T02:48:29.518 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.533+0000 7f4ace7fc700 1 -- 172.21.15.105:0/1292487942 <== mgr.14221 v2:172.21.15.105:6800/2334550011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2820 (secure 0 0 0) 0x7f4ad8068b20 con 0x7f4ac406c4e0 2024-04-06T02:48:29.518 INFO:teuthology.orchestra.run.smithi105.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T02:48:29.518 INFO:teuthology.orchestra.run.smithi105.stdout:alertmanager.smithi105 smithi105 *:9093,9094 running (3m) 2m ago 5m 26.7M - 0.25.0 c8568f914cd2 5b927ee87a96 2024-04-06T02:48:29.518 INFO:teuthology.orchestra.run.smithi105.stdout:ceph-exporter.smithi105 smithi105 running (5m) 2m ago 5m 7084k - 18.2.1 5be31c24972a 448b72643fea 2024-04-06T02:48:29.518 INFO:teuthology.orchestra.run.smithi105.stdout:ceph-exporter.smithi184 smithi184 running (4m) 65s ago 4m 6966k - 18.2.1 5be31c24972a 52fd408df95f 2024-04-06T02:48:29.519 INFO:teuthology.orchestra.run.smithi105.stdout:crash.smithi105 smithi105 running (5m) 2m ago 5m 6660k - 18.2.1 5be31c24972a 8e21a7fab0b2 2024-04-06T02:48:29.519 INFO:teuthology.orchestra.run.smithi105.stdout:crash.smithi184 smithi184 running (4m) 65s ago 4m 6660k - 18.2.1 5be31c24972a a0c065bc6908 2024-04-06T02:48:29.519 INFO:teuthology.orchestra.run.smithi105.stdout:grafana.smithi105 smithi105 *:3000 running (3m) 2m ago 5m 78.0M - 9.4.7 954c08fa6188 413a7bf83436 2024-04-06T02:48:29.519 INFO:teuthology.orchestra.run.smithi105.stdout:mgr.smithi105.vqqcpx smithi105 *:9283,8765,8443 running (6m) 2m ago 6m 480M - 18.2.1 5be31c24972a 07c0a110a1b8 2024-04-06T02:48:29.519 INFO:teuthology.orchestra.run.smithi105.stdout:mgr.smithi184.fzhhql smithi184 *:8443,9283,8765 running (4m) 65s ago 4m 442M - 18.2.1 5be31c24972a a09137f55e51 2024-04-06T02:48:29.519 INFO:teuthology.orchestra.run.smithi105.stdout:mon.smithi105 smithi105 running (6m) 2m ago 6m 44.9M 2048M 18.2.1 5be31c24972a d27bf0498159 2024-04-06T02:48:29.519 INFO:teuthology.orchestra.run.smithi105.stdout:mon.smithi184 smithi184 running (4m) 65s ago 4m 44.0M 2048M 18.2.1 5be31c24972a 9054b877f58f 2024-04-06T02:48:29.519 INFO:teuthology.orchestra.run.smithi105.stdout:node-exporter.smithi105 smithi105 *:9100 running (5m) 2m ago 5m 15.6M - 1.5.0 0da6a335fe13 543e8799de93 2024-04-06T02:48:29.519 INFO:teuthology.orchestra.run.smithi105.stdout:node-exporter.smithi184 smithi184 *:9100 running (4m) 65s ago 4m 15.8M - 1.5.0 0da6a335fe13 7688716a6f4a 2024-04-06T02:48:29.519 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0 smithi105 running (3m) 2m ago 3m 37.7M 3946M 18.2.1 5be31c24972a 339c3f865ef9 2024-04-06T02:48:29.519 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1 smithi105 running (2m) 2m ago 2m 39.3M 3946M 18.2.1 5be31c24972a eb4cf9ea6b1b 2024-04-06T02:48:29.519 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2 smithi105 running (2m) 2m ago 2m 37.0M 3946M 18.2.1 5be31c24972a 0392f09bdb6b 2024-04-06T02:48:29.520 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3 smithi184 running (2m) 65s ago 2m 41.4M 4970M 18.2.1 5be31c24972a e833150f2893 2024-04-06T02:48:29.520 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4 smithi184 running (96s) 65s ago 95s 39.6M 4970M 18.2.1 5be31c24972a a13051e8d0fb 2024-04-06T02:48:29.520 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5 smithi184 running (70s) 65s ago 70s 14.0M 4970M 18.2.1 5be31c24972a b7b1ce8932de 2024-04-06T02:48:29.520 INFO:teuthology.orchestra.run.smithi105.stdout:prometheus.smithi105 smithi105 *:9095 running (3m) 2m ago 4m 40.1M - 2.43.0 a07b618ecd1d ee5967404456 2024-04-06T02:48:29.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.535+0000 7f4adf55e700 1 -- 172.21.15.105:0/1292487942 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f4ac406c4e0 msgr2=0x7f4ac406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:29.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.535+0000 7f4adf55e700 1 --2- 172.21.15.105:0/1292487942 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f4ac406c4e0 0x7f4ac406e9a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8006010 tx=0x7f4ac800b540 comp rx=0 tx=0).stop 2024-04-06T02:48:29.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.535+0000 7f4adf55e700 1 -- 172.21.15.105:0/1292487942 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ad8100cb0 msgr2=0x7f4ad8196930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:29.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.535+0000 7f4adf55e700 1 --2- 172.21.15.105:0/1292487942 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ad8100cb0 0x7f4ad8196930 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4ad400b4b0 tx=0x7f4ad400b7c0 comp rx=0 tx=0).stop 2024-04-06T02:48:29.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.535+0000 7f4adf55e700 1 -- 172.21.15.105:0/1292487942 shutdown_connections 2024-04-06T02:48:29.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.535+0000 7f4adf55e700 1 --2- 172.21.15.105:0/1292487942 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f4ac406c4e0 0x7f4ac406e9a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:29.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.535+0000 7f4adf55e700 1 --2- 172.21.15.105:0/1292487942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4ad80ffab0 0x7f4ad81963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:29.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.535+0000 7f4adf55e700 1 --2- 172.21.15.105:0/1292487942 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ad8100cb0 0x7f4ad8196930 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:29.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.535+0000 7f4adf55e700 1 -- 172.21.15.105:0/1292487942 >> 172.21.15.105:0/1292487942 conn(0x7f4ad80fb030 msgr2=0x7f4ad8068410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:29.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.535+0000 7f4adf55e700 1 -- 172.21.15.105:0/1292487942 shutdown_connections 2024-04-06T02:48:29.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:29.535+0000 7f4adf55e700 1 -- 172.21.15.105:0/1292487942 wait complete. 2024-04-06T02:48:30.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:29 smithi105 ceph-mon[29228]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:30.213 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch ls' 2024-04-06T02:48:30.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:29 smithi184 ceph-mon[37439]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:30.418 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:48:31.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:30 smithi105 ceph-mon[29228]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:48:31.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:30 smithi184 ceph-mon[37439]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:48:32.073 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.086+0000 7ff7a4e6a700 1 -- 172.21.15.105:0/2942993132 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7a0100be0 msgr2=0x7ff7a0101000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:32.073 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.086+0000 7ff7a4e6a700 1 --2- 172.21.15.105:0/2942993132 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7a0100be0 0x7ff7a0101000 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff790009910 tx=0x7ff790009c20 comp rx=0 tx=0).stop 2024-04-06T02:48:32.073 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.086+0000 7ff7a4e6a700 1 -- 172.21.15.105:0/2942993132 shutdown_connections 2024-04-06T02:48:32.073 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.086+0000 7ff7a4e6a700 1 --2- 172.21.15.105:0/2942993132 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff7a0101de0 0x7ff7a0102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:32.073 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.086+0000 7ff7a4e6a700 1 --2- 172.21.15.105:0/2942993132 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7a0100be0 0x7ff7a0101000 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:32.073 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.086+0000 7ff7a4e6a700 1 -- 172.21.15.105:0/2942993132 >> 172.21.15.105:0/2942993132 conn(0x7ff7a00fc140 msgr2=0x7ff7a00fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:32.074 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.086+0000 7ff7a4e6a700 1 -- 172.21.15.105:0/2942993132 shutdown_connections 2024-04-06T02:48:32.074 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.086+0000 7ff7a4e6a700 1 -- 172.21.15.105:0/2942993132 wait complete. 2024-04-06T02:48:32.074 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.086+0000 7ff7a4e6a700 1 Processor -- start 2024-04-06T02:48:32.074 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.086+0000 7ff7a4e6a700 1 -- start start 2024-04-06T02:48:32.074 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.087+0000 7ff7a4e6a700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff7a0100be0 0x7ff7a0198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:32.074 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.087+0000 7ff7a4e6a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7a0101de0 0x7ff7a0198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:32.074 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.087+0000 7ff7a4e6a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a01990e0 con 0x7ff7a0101de0 2024-04-06T02:48:32.074 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.087+0000 7ff7a4e6a700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a0199220 con 0x7ff7a0100be0 2024-04-06T02:48:32.074 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.087+0000 7ff79dd9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7a0101de0 0x7ff7a0198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:32.074 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.087+0000 7ff79dd9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7a0101de0 0x7ff7a0198ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33732/0 (socket says 172.21.15.105:33732) 2024-04-06T02:48:32.074 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.087+0000 7ff79dd9b700 1 -- 172.21.15.105:0/357720814 learned_addr learned my addr 172.21.15.105:0/357720814 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:48:32.074 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.087+0000 7ff79e59c700 1 --2- 172.21.15.105:0/357720814 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff7a0100be0 0x7ff7a0198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:32.075 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.087+0000 7ff79dd9b700 1 -- 172.21.15.105:0/357720814 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff7a0100be0 msgr2=0x7ff7a0198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:32.075 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.087+0000 7ff79dd9b700 1 --2- 172.21.15.105:0/357720814 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff7a0100be0 0x7ff7a0198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:32.075 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.087+0000 7ff79dd9b700 1 -- 172.21.15.105:0/357720814 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7900095f0 con 0x7ff7a0101de0 2024-04-06T02:48:32.075 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.087+0000 7ff79dd9b700 1 --2- 172.21.15.105:0/357720814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7a0101de0 0x7ff7a0198ac0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff79400d8d0 tx=0x7ff79400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:32.075 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.087+0000 7ff78f7fe700 1 -- 172.21.15.105:0/357720814 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7940098a0 con 0x7ff7a0101de0 2024-04-06T02:48:32.075 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.088+0000 7ff78f7fe700 1 -- 172.21.15.105:0/357720814 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff794010460 con 0x7ff7a0101de0 2024-04-06T02:48:32.075 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.088+0000 7ff7a4e6a700 1 -- 172.21.15.105:0/357720814 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7a019dcd0 con 0x7ff7a0101de0 2024-04-06T02:48:32.075 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.088+0000 7ff78f7fe700 1 -- 172.21.15.105:0/357720814 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff79400f5f0 con 0x7ff7a0101de0 2024-04-06T02:48:32.075 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.088+0000 7ff7a4e6a700 1 -- 172.21.15.105:0/357720814 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7a019e220 con 0x7ff7a0101de0 2024-04-06T02:48:32.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.088+0000 7ff7a4e6a700 1 -- 172.21.15.105:0/357720814 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7a0105ec0 con 0x7ff7a0101de0 2024-04-06T02:48:32.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.089+0000 7ff78f7fe700 1 -- 172.21.15.105:0/357720814 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7ff794009a00 con 0x7ff7a0101de0 2024-04-06T02:48:32.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.089+0000 7ff78f7fe700 1 --2- 172.21.15.105:0/357720814 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7ff78806c4e0 0x7ff78806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:32.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.089+0000 7ff79e59c700 1 --2- 172.21.15.105:0/357720814 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7ff78806c4e0 0x7ff78806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:32.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.089+0000 7ff78f7fe700 1 -- 172.21.15.105:0/357720814 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff79408b390 con 0x7ff7a0101de0 2024-04-06T02:48:32.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.090+0000 7ff79e59c700 1 --2- 172.21.15.105:0/357720814 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7ff78806c4e0 0x7ff78806e9a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff790006010 tx=0x7ff79000b540 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:32.079 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.094+0000 7ff78f7fe700 1 -- 172.21.15.105:0/357720814 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff794059940 con 0x7ff7a0101de0 2024-04-06T02:48:32.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:31 smithi105 ceph-mon[29228]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:32.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:31 smithi184 ceph-mon[37439]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:32.275 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.290+0000 7ff7a4e6a700 1 -- 172.21.15.105:0/357720814 --> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7a00611d0 con 0x7ff78806c4e0 2024-04-06T02:48:32.280 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.294+0000 7ff78f7fe700 1 -- 172.21.15.105:0/357720814 <== mgr.14221 v2:172.21.15.105:6800/2334550011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1310 (secure 0 0 0) 0x7ff7a00611d0 con 0x7ff78806c4e0 2024-04-06T02:48:32.280 INFO:teuthology.orchestra.run.smithi105.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-06T02:48:32.280 INFO:teuthology.orchestra.run.smithi105.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-04-06T02:48:32.280 INFO:teuthology.orchestra.run.smithi105.stdout:ceph-exporter 2/2 2m ago 6m * 2024-04-06T02:48:32.280 INFO:teuthology.orchestra.run.smithi105.stdout:crash 2/2 2m ago 6m * 2024-04-06T02:48:32.280 INFO:teuthology.orchestra.run.smithi105.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-04-06T02:48:32.280 INFO:teuthology.orchestra.run.smithi105.stdout:mgr 2/2 2m ago 6m count:2 2024-04-06T02:48:32.280 INFO:teuthology.orchestra.run.smithi105.stdout:mon 2/2 2m ago 5m smithi105:172.21.15.105=smithi105;smithi184:172.21.15.184=smithi184;count:2 2024-04-06T02:48:32.281 INFO:teuthology.orchestra.run.smithi105.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-04-06T02:48:32.281 INFO:teuthology.orchestra.run.smithi105.stdout:osd 6 2m ago - 2024-04-06T02:48:32.281 INFO:teuthology.orchestra.run.smithi105.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-04-06T02:48:32.282 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.297+0000 7ff7a4e6a700 1 -- 172.21.15.105:0/357720814 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7ff78806c4e0 msgr2=0x7ff78806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:32.282 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.297+0000 7ff7a4e6a700 1 --2- 172.21.15.105:0/357720814 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7ff78806c4e0 0x7ff78806e9a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff790006010 tx=0x7ff79000b540 comp rx=0 tx=0).stop 2024-04-06T02:48:32.282 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.297+0000 7ff7a4e6a700 1 -- 172.21.15.105:0/357720814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7a0101de0 msgr2=0x7ff7a0198ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:32.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.297+0000 7ff7a4e6a700 1 --2- 172.21.15.105:0/357720814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7a0101de0 0x7ff7a0198ac0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff79400d8d0 tx=0x7ff79400dbe0 comp rx=0 tx=0).stop 2024-04-06T02:48:32.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.297+0000 7ff7a4e6a700 1 -- 172.21.15.105:0/357720814 shutdown_connections 2024-04-06T02:48:32.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.297+0000 7ff7a4e6a700 1 --2- 172.21.15.105:0/357720814 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7ff78806c4e0 0x7ff78806e9a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:32.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.297+0000 7ff7a4e6a700 1 --2- 172.21.15.105:0/357720814 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff7a0100be0 0x7ff7a0198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:32.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.297+0000 7ff7a4e6a700 1 --2- 172.21.15.105:0/357720814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7a0101de0 0x7ff7a0198ac0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:32.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.297+0000 7ff7a4e6a700 1 -- 172.21.15.105:0/357720814 >> 172.21.15.105:0/357720814 conn(0x7ff7a00fc140 msgr2=0x7ff7a0103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:32.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.297+0000 7ff7a4e6a700 1 -- 172.21.15.105:0/357720814 shutdown_connections 2024-04-06T02:48:32.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:32.297+0000 7ff7a4e6a700 1 -- 172.21.15.105:0/357720814 wait complete. 2024-04-06T02:48:32.920 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch host ls' 2024-04-06T02:48:33.127 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:48:33.574 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:33 smithi105 ceph-mon[29228]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:48:33.574 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:33 smithi105 ceph-mon[29228]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:33.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:33 smithi184 ceph-mon[37439]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:48:33.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:33 smithi184 ceph-mon[37439]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:34.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.722+0000 7fee847e0700 1 -- 172.21.15.105:0/1210931645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee7c100bc0 msgr2=0x7fee7c100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:34.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.722+0000 7fee847e0700 1 --2- 172.21.15.105:0/1210931645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee7c100bc0 0x7fee7c100fe0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fee78009910 tx=0x7fee78009c20 comp rx=0 tx=0).stop 2024-04-06T02:48:34.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.722+0000 7fee847e0700 1 -- 172.21.15.105:0/1210931645 shutdown_connections 2024-04-06T02:48:34.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.722+0000 7fee847e0700 1 --2- 172.21.15.105:0/1210931645 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee7c101dc0 0x7fee7c102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:34.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.722+0000 7fee847e0700 1 --2- 172.21.15.105:0/1210931645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee7c100bc0 0x7fee7c100fe0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:34.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.722+0000 7fee847e0700 1 -- 172.21.15.105:0/1210931645 >> 172.21.15.105:0/1210931645 conn(0x7fee7c0fc140 msgr2=0x7fee7c0fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:34.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.722+0000 7fee847e0700 1 -- 172.21.15.105:0/1210931645 shutdown_connections 2024-04-06T02:48:34.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.722+0000 7fee847e0700 1 -- 172.21.15.105:0/1210931645 wait complete. 2024-04-06T02:48:34.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.723+0000 7fee847e0700 1 Processor -- start 2024-04-06T02:48:34.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.723+0000 7fee847e0700 1 -- start start 2024-04-06T02:48:34.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.723+0000 7fee847e0700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee7c100bc0 0x7fee7c071da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:34.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.723+0000 7fee847e0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee7c101dc0 0x7fee7c0722e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:34.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.723+0000 7fee847e0700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee7c072900 con 0x7fee7c101dc0 2024-04-06T02:48:34.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.723+0000 7fee847e0700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee7c072a40 con 0x7fee7c100bc0 2024-04-06T02:48:34.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.723+0000 7fee81d7b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee7c101dc0 0x7fee7c0722e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:34.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.723+0000 7fee81d7b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee7c101dc0 0x7fee7c0722e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33748/0 (socket says 172.21.15.105:33748) 2024-04-06T02:48:34.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.723+0000 7fee81d7b700 1 -- 172.21.15.105:0/473162325 learned_addr learned my addr 172.21.15.105:0/473162325 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:48:34.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.723+0000 7fee8257c700 1 --2- 172.21.15.105:0/473162325 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee7c100bc0 0x7fee7c071da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:34.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.723+0000 7fee81d7b700 1 -- 172.21.15.105:0/473162325 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee7c100bc0 msgr2=0x7fee7c071da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:34.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.723+0000 7fee81d7b700 1 --2- 172.21.15.105:0/473162325 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee7c100bc0 0x7fee7c071da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:34.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.723+0000 7fee81d7b700 1 -- 172.21.15.105:0/473162325 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee780095f0 con 0x7fee7c101dc0 2024-04-06T02:48:34.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.723+0000 7fee81d7b700 1 --2- 172.21.15.105:0/473162325 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee7c101dc0 0x7fee7c0722e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fee6c00c8a0 tx=0x7fee6c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:34.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.723+0000 7fee737fe700 1 -- 172.21.15.105:0/473162325 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee6c0078e0 con 0x7fee7c101dc0 2024-04-06T02:48:34.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.723+0000 7fee737fe700 1 -- 172.21.15.105:0/473162325 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fee6c00f450 con 0x7fee7c101dc0 2024-04-06T02:48:34.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.723+0000 7fee847e0700 1 -- 172.21.15.105:0/473162325 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee7c0734d0 con 0x7fee7c101dc0 2024-04-06T02:48:34.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.723+0000 7fee737fe700 1 -- 172.21.15.105:0/473162325 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee6c00e5e0 con 0x7fee7c101dc0 2024-04-06T02:48:34.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.724+0000 7fee847e0700 1 -- 172.21.15.105:0/473162325 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee7c073a20 con 0x7fee7c101dc0 2024-04-06T02:48:34.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.724+0000 7fee847e0700 1 -- 172.21.15.105:0/473162325 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee7c109720 con 0x7fee7c101dc0 2024-04-06T02:48:34.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.725+0000 7fee737fe700 1 -- 172.21.15.105:0/473162325 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fee6c00f5c0 con 0x7fee7c101dc0 2024-04-06T02:48:34.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.725+0000 7fee737fe700 1 --2- 172.21.15.105:0/473162325 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fee6806c4e0 0x7fee6806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:34.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.725+0000 7fee737fe700 1 -- 172.21.15.105:0/473162325 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fee6c08b360 con 0x7fee7c101dc0 2024-04-06T02:48:34.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.725+0000 7fee8257c700 1 --2- 172.21.15.105:0/473162325 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fee6806c4e0 0x7fee6806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:34.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.726+0000 7fee8257c700 1 --2- 172.21.15.105:0/473162325 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fee6806c4e0 0x7fee6806e9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fee78006010 tx=0x7fee78005280 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:34.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.730+0000 7fee737fe700 1 -- 172.21.15.105:0/473162325 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fee6c059990 con 0x7fee7c101dc0 2024-04-06T02:48:34.908 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.922+0000 7fee847e0700 1 -- 172.21.15.105:0/473162325 --> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fee7c106710 con 0x7fee6806c4e0 2024-04-06T02:48:34.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.924+0000 7fee737fe700 1 -- 172.21.15.105:0/473162325 <== mgr.14221 v2:172.21.15.105:6800/2334550011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7fee7c106710 con 0x7fee6806c4e0 2024-04-06T02:48:34.909 INFO:teuthology.orchestra.run.smithi105.stdout:HOST ADDR LABELS STATUS 2024-04-06T02:48:34.909 INFO:teuthology.orchestra.run.smithi105.stdout:smithi105 172.21.15.105 2024-04-06T02:48:34.909 INFO:teuthology.orchestra.run.smithi105.stdout:smithi184 172.21.15.184 2024-04-06T02:48:34.909 INFO:teuthology.orchestra.run.smithi105.stdout:2 hosts in cluster 2024-04-06T02:48:34.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.926+0000 7fee847e0700 1 -- 172.21.15.105:0/473162325 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fee6806c4e0 msgr2=0x7fee6806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:34.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.926+0000 7fee847e0700 1 --2- 172.21.15.105:0/473162325 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fee6806c4e0 0x7fee6806e9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fee78006010 tx=0x7fee78005280 comp rx=0 tx=0).stop 2024-04-06T02:48:34.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.926+0000 7fee847e0700 1 -- 172.21.15.105:0/473162325 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee7c101dc0 msgr2=0x7fee7c0722e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:34.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.926+0000 7fee847e0700 1 --2- 172.21.15.105:0/473162325 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee7c101dc0 0x7fee7c0722e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fee6c00c8a0 tx=0x7fee6c00cbb0 comp rx=0 tx=0).stop 2024-04-06T02:48:34.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.926+0000 7fee847e0700 1 -- 172.21.15.105:0/473162325 shutdown_connections 2024-04-06T02:48:34.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.926+0000 7fee847e0700 1 --2- 172.21.15.105:0/473162325 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fee6806c4e0 0x7fee6806e9a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:34.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.926+0000 7fee847e0700 1 --2- 172.21.15.105:0/473162325 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee7c100bc0 0x7fee7c071da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:34.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.926+0000 7fee847e0700 1 --2- 172.21.15.105:0/473162325 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee7c101dc0 0x7fee7c0722e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:34.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.926+0000 7fee847e0700 1 -- 172.21.15.105:0/473162325 >> 172.21.15.105:0/473162325 conn(0x7fee7c0fc140 msgr2=0x7fee7c104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:34.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.926+0000 7fee847e0700 1 -- 172.21.15.105:0/473162325 shutdown_connections 2024-04-06T02:48:34.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:34.926+0000 7fee847e0700 1 -- 172.21.15.105:0/473162325 wait complete. 2024-04-06T02:48:35.558 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch device ls' 2024-04-06T02:48:35.764 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:48:35.885 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:35 smithi105 ceph-mon[29228]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:35.885 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:35 smithi105 ceph-mon[29228]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:48:35.919 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:35 smithi184 ceph-mon[37439]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:35.920 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:35 smithi184 ceph-mon[37439]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:48:37.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.392+0000 7f02965fc700 1 -- 172.21.15.105:0/1114526000 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0290100bc0 msgr2=0x7f0290100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:37.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.393+0000 7f02965fc700 1 --2- 172.21.15.105:0/1114526000 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0290100bc0 0x7f0290100fe0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0280009910 tx=0x7f0280009c20 comp rx=0 tx=0).stop 2024-04-06T02:48:37.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.393+0000 7f02965fc700 1 -- 172.21.15.105:0/1114526000 shutdown_connections 2024-04-06T02:48:37.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.393+0000 7f02965fc700 1 --2- 172.21.15.105:0/1114526000 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0290101dc0 0x7f0290102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:37.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.393+0000 7f02965fc700 1 --2- 172.21.15.105:0/1114526000 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0290100bc0 0x7f0290100fe0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:37.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.393+0000 7f02965fc700 1 -- 172.21.15.105:0/1114526000 >> 172.21.15.105:0/1114526000 conn(0x7f02900fc140 msgr2=0x7f02900fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:37.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.393+0000 7f02965fc700 1 -- 172.21.15.105:0/1114526000 shutdown_connections 2024-04-06T02:48:37.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.393+0000 7f02965fc700 1 -- 172.21.15.105:0/1114526000 wait complete. 2024-04-06T02:48:37.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.393+0000 7f02965fc700 1 Processor -- start 2024-04-06T02:48:37.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.393+0000 7f02965fc700 1 -- start start 2024-04-06T02:48:37.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.393+0000 7f02965fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0290100bc0 0x7f0290196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:37.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.393+0000 7f02965fc700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0290101dc0 0x7f02901968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:37.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.393+0000 7f02965fc700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0290196ef0 con 0x7f0290100bc0 2024-04-06T02:48:37.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.393+0000 7f02965fc700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0290197030 con 0x7f0290101dc0 2024-04-06T02:48:37.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.394+0000 7f028ffff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0290100bc0 0x7f0290196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:37.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.394+0000 7f028ffff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0290100bc0 0x7f0290196390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33762/0 (socket says 172.21.15.105:33762) 2024-04-06T02:48:37.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.394+0000 7f028ffff700 1 -- 172.21.15.105:0/1222210865 learned_addr learned my addr 172.21.15.105:0/1222210865 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:48:37.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.394+0000 7f028ffff700 1 -- 172.21.15.105:0/1222210865 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0290101dc0 msgr2=0x7f02901968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:37.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.394+0000 7f028ffff700 1 --2- 172.21.15.105:0/1222210865 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0290101dc0 0x7f02901968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:37.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.394+0000 7f028ffff700 1 -- 172.21.15.105:0/1222210865 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02800095f0 con 0x7f0290100bc0 2024-04-06T02:48:37.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.394+0000 7f028ffff700 1 --2- 172.21.15.105:0/1222210865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0290100bc0 0x7f0290196390 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f028000bda0 tx=0x7f028000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:37.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.394+0000 7f028d7fa700 1 -- 172.21.15.105:0/1222210865 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f028001d070 con 0x7f0290100bc0 2024-04-06T02:48:37.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.394+0000 7f028d7fa700 1 -- 172.21.15.105:0/1222210865 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f028000f460 con 0x7f0290100bc0 2024-04-06T02:48:37.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.394+0000 7f02965fc700 1 -- 172.21.15.105:0/1222210865 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f029019ba80 con 0x7f0290100bc0 2024-04-06T02:48:37.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.394+0000 7f028d7fa700 1 -- 172.21.15.105:0/1222210865 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0280021640 con 0x7f0290100bc0 2024-04-06T02:48:37.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.394+0000 7f02965fc700 1 -- 172.21.15.105:0/1222210865 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f029019bf70 con 0x7f0290100bc0 2024-04-06T02:48:37.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.395+0000 7f02965fc700 1 -- 172.21.15.105:0/1222210865 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0290109720 con 0x7f0290100bc0 2024-04-06T02:48:37.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.396+0000 7f028d7fa700 1 -- 172.21.15.105:0/1222210865 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f028000f5d0 con 0x7f0290100bc0 2024-04-06T02:48:37.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.396+0000 7f028d7fa700 1 --2- 172.21.15.105:0/1222210865 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f027806c4e0 0x7f027806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:37.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.396+0000 7f028d7fa700 1 -- 172.21.15.105:0/1222210865 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f028008c770 con 0x7f0290100bc0 2024-04-06T02:48:37.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.396+0000 7f028f7fe700 1 --2- 172.21.15.105:0/1222210865 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f027806c4e0 0x7f027806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:37.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.396+0000 7f028f7fe700 1 --2- 172.21.15.105:0/1222210865 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f027806c4e0 0x7f027806e9a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0284005d90 tx=0x7f0284005d00 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:37.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.400+0000 7f028d7fa700 1 -- 172.21.15.105:0/1222210865 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f028005ae40 con 0x7f0290100bc0 2024-04-06T02:48:37.589 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.603+0000 7f02965fc700 1 -- 172.21.15.105:0/1222210865 --> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f0290106710 con 0x7f027806c4e0 2024-04-06T02:48:37.590 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.605+0000 7f028d7fa700 1 -- 172.21.15.105:0/1222210865 <== mgr.14221 v2:172.21.15.105:6800/2334550011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f0290106710 con 0x7f027806c4e0 2024-04-06T02:48:37.590 INFO:teuthology.orchestra.run.smithi105.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-06T02:48:37.590 INFO:teuthology.orchestra.run.smithi105.stdout:smithi105 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400TR400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-06T02:48:37.591 INFO:teuthology.orchestra.run.smithi105.stdout:smithi184 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT626500CE400BGN 372G No 70s ago Has a FileSystem, LVM detected 2024-04-06T02:48:37.592 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.607+0000 7f02965fc700 1 -- 172.21.15.105:0/1222210865 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f027806c4e0 msgr2=0x7f027806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:37.592 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.607+0000 7f02965fc700 1 --2- 172.21.15.105:0/1222210865 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f027806c4e0 0x7f027806e9a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0284005d90 tx=0x7f0284005d00 comp rx=0 tx=0).stop 2024-04-06T02:48:37.592 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.607+0000 7f02965fc700 1 -- 172.21.15.105:0/1222210865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0290100bc0 msgr2=0x7f0290196390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:37.592 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.607+0000 7f02965fc700 1 --2- 172.21.15.105:0/1222210865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0290100bc0 0x7f0290196390 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f028000bda0 tx=0x7f028000be80 comp rx=0 tx=0).stop 2024-04-06T02:48:37.593 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.607+0000 7f02965fc700 1 -- 172.21.15.105:0/1222210865 shutdown_connections 2024-04-06T02:48:37.593 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.607+0000 7f02965fc700 1 --2- 172.21.15.105:0/1222210865 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f027806c4e0 0x7f027806e9a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:37.593 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.607+0000 7f02965fc700 1 --2- 172.21.15.105:0/1222210865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0290100bc0 0x7f0290196390 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:37.593 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.607+0000 7f02965fc700 1 --2- 172.21.15.105:0/1222210865 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0290101dc0 0x7f02901968d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:37.593 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.607+0000 7f02965fc700 1 -- 172.21.15.105:0/1222210865 >> 172.21.15.105:0/1222210865 conn(0x7f02900fc140 msgr2=0x7f0290104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:37.593 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.607+0000 7f02965fc700 1 -- 172.21.15.105:0/1222210865 shutdown_connections 2024-04-06T02:48:37.593 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:37.607+0000 7f02965fc700 1 -- 172.21.15.105:0/1222210865 wait complete. 2024-04-06T02:48:37.913 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:37 smithi105 ceph-mon[29228]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:37.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:37 smithi184 ceph-mon[37439]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:38.245 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-06T02:48:38.255 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi105.front.sepia.ceph.com 2024-04-06T02:48:38.255 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-04-06T02:48:38.459 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:48:38.902 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:38 smithi105 ceph-mon[29228]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:48:38.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:38 smithi184 ceph-mon[37439]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:48:39.909 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:39 smithi105 ceph-mon[29228]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:39.910 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:39 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:48:39.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:39 smithi184 ceph-mon[37439]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:39.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:39 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:48:40.108 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.120+0000 7f5b386aa700 1 -- 172.21.15.105:0/1367708394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b300ffad0 msgr2=0x7f5b300ffef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:40.108 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.120+0000 7f5b386aa700 1 --2- 172.21.15.105:0/1367708394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b300ffad0 0x7f5b300ffef0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5b20009910 tx=0x7f5b20009c20 comp rx=0 tx=0).stop 2024-04-06T02:48:40.108 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.120+0000 7f5b386aa700 1 -- 172.21.15.105:0/1367708394 shutdown_connections 2024-04-06T02:48:40.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.120+0000 7f5b386aa700 1 --2- 172.21.15.105:0/1367708394 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b30100cd0 0x7f5b30101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:40.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.120+0000 7f5b386aa700 1 --2- 172.21.15.105:0/1367708394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b300ffad0 0x7f5b300ffef0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:40.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.120+0000 7f5b386aa700 1 -- 172.21.15.105:0/1367708394 >> 172.21.15.105:0/1367708394 conn(0x7f5b300fb050 msgr2=0x7f5b300fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:40.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.120+0000 7f5b386aa700 1 -- 172.21.15.105:0/1367708394 shutdown_connections 2024-04-06T02:48:40.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.120+0000 7f5b386aa700 1 -- 172.21.15.105:0/1367708394 wait complete. 2024-04-06T02:48:40.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.121+0000 7f5b386aa700 1 Processor -- start 2024-04-06T02:48:40.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.121+0000 7f5b386aa700 1 -- start start 2024-04-06T02:48:40.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.121+0000 7f5b386aa700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b300ffad0 0x7f5b300698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:40.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.121+0000 7f5b386aa700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b30100cd0 0x7f5b3019b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:40.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.121+0000 7f5b386aa700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3019b6e0 con 0x7f5b300ffad0 2024-04-06T02:48:40.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.121+0000 7f5b386aa700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3019b820 con 0x7f5b30100cd0 2024-04-06T02:48:40.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.121+0000 7f5b36446700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b300ffad0 0x7f5b300698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:40.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.121+0000 7f5b36446700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b300ffad0 0x7f5b300698f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33776/0 (socket says 172.21.15.105:33776) 2024-04-06T02:48:40.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.121+0000 7f5b36446700 1 -- 172.21.15.105:0/3018423362 learned_addr learned my addr 172.21.15.105:0/3018423362 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:48:40.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.121+0000 7f5b35c45700 1 --2- 172.21.15.105:0/3018423362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b30100cd0 0x7f5b3019b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:40.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.121+0000 7f5b36446700 1 -- 172.21.15.105:0/3018423362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b30100cd0 msgr2=0x7f5b3019b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:40.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.121+0000 7f5b36446700 1 --2- 172.21.15.105:0/3018423362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b30100cd0 0x7f5b3019b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:40.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.122+0000 7f5b36446700 1 -- 172.21.15.105:0/3018423362 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b200095f0 con 0x7f5b300ffad0 2024-04-06T02:48:40.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.122+0000 7f5b35c45700 1 --2- 172.21.15.105:0/3018423362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b30100cd0 0x7f5b3019b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T02:48:40.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.122+0000 7f5b36446700 1 --2- 172.21.15.105:0/3018423362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b300ffad0 0x7f5b300698f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5b2000bda0 tx=0x7f5b2000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:40.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.122+0000 7f5b277fe700 1 -- 172.21.15.105:0/3018423362 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b2001d070 con 0x7f5b300ffad0 2024-04-06T02:48:40.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.122+0000 7f5b386aa700 1 -- 172.21.15.105:0/3018423362 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b3019baa0 con 0x7f5b300ffad0 2024-04-06T02:48:40.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.122+0000 7f5b277fe700 1 -- 172.21.15.105:0/3018423362 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5b2000f460 con 0x7f5b300ffad0 2024-04-06T02:48:40.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.122+0000 7f5b277fe700 1 -- 172.21.15.105:0/3018423362 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b20021640 con 0x7f5b300ffad0 2024-04-06T02:48:40.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.122+0000 7f5b386aa700 1 -- 172.21.15.105:0/3018423362 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b3019bf90 con 0x7f5b300ffad0 2024-04-06T02:48:40.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.123+0000 7f5b386aa700 1 -- 172.21.15.105:0/3018423362 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b30108630 con 0x7f5b300ffad0 2024-04-06T02:48:40.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.123+0000 7f5b277fe700 1 -- 172.21.15.105:0/3018423362 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f5b2000fa00 con 0x7f5b300ffad0 2024-04-06T02:48:40.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.124+0000 7f5b277fe700 1 --2- 172.21.15.105:0/3018423362 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f5b1c06c4e0 0x7f5b1c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:40.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.124+0000 7f5b277fe700 1 -- 172.21.15.105:0/3018423362 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f5b2008c7d0 con 0x7f5b300ffad0 2024-04-06T02:48:40.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.124+0000 7f5b35c45700 1 --2- 172.21.15.105:0/3018423362 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f5b1c06c4e0 0x7f5b1c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:40.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.124+0000 7f5b35c45700 1 --2- 172.21.15.105:0/3018423362 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f5b1c06c4e0 0x7f5b1c06e9a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c005950 tx=0x7f5b2c0058e0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:40.113 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.128+0000 7f5b277fe700 1 -- 172.21.15.105:0/3018423362 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5b2005ad80 con 0x7f5b300ffad0 2024-04-06T02:48:40.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:40.342+0000 7f5b386aa700 1 -- 172.21.15.105:0/3018423362 --> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b30105620 con 0x7f5b1c06c4e0 2024-04-06T02:48:40.958 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:40 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-06T02:48:41.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:40 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-06T02:48:41.714 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:41.729+0000 7f5b277fe700 1 -- 172.21.15.105:0/3018423362 <== mgr.14221 v2:172.21.15.105:6800/2334550011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5b30105620 con 0x7f5b1c06c4e0 2024-04-06T02:48:41.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:41.733+0000 7f5b386aa700 1 -- 172.21.15.105:0/3018423362 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f5b1c06c4e0 msgr2=0x7f5b1c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:41.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:41.733+0000 7f5b386aa700 1 --2- 172.21.15.105:0/3018423362 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f5b1c06c4e0 0x7f5b1c06e9a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c005950 tx=0x7f5b2c0058e0 comp rx=0 tx=0).stop 2024-04-06T02:48:41.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:41.733+0000 7f5b386aa700 1 -- 172.21.15.105:0/3018423362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b300ffad0 msgr2=0x7f5b300698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:41.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:41.733+0000 7f5b386aa700 1 --2- 172.21.15.105:0/3018423362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b300ffad0 0x7f5b300698f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5b2000bda0 tx=0x7f5b2000be80 comp rx=0 tx=0).stop 2024-04-06T02:48:41.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:41.733+0000 7f5b386aa700 1 -- 172.21.15.105:0/3018423362 shutdown_connections 2024-04-06T02:48:41.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:41.733+0000 7f5b386aa700 1 --2- 172.21.15.105:0/3018423362 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f5b1c06c4e0 0x7f5b1c06e9a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:41.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:41.733+0000 7f5b386aa700 1 --2- 172.21.15.105:0/3018423362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b300ffad0 0x7f5b300698f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:41.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:41.733+0000 7f5b386aa700 1 --2- 172.21.15.105:0/3018423362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b30100cd0 0x7f5b3019b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:41.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:41.733+0000 7f5b386aa700 1 -- 172.21.15.105:0/3018423362 >> 172.21.15.105:0/3018423362 conn(0x7f5b300fb050 msgr2=0x7f5b30103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:41.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:41.733+0000 7f5b386aa700 1 -- 172.21.15.105:0/3018423362 shutdown_connections 2024-04-06T02:48:41.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:41.733+0000 7f5b386aa700 1 -- 172.21.15.105:0/3018423362 wait complete. 2024-04-06T02:48:41.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:41 smithi184 ceph-mon[37439]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:48:41.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:41 smithi184 ceph-mon[37439]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:41.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:41 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-06T02:48:41.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:41 smithi184 ceph-mon[37439]: osdmap e42: 6 total, 6 up, 6 in 2024-04-06T02:48:41.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:41 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-06T02:48:41.999 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:41 smithi105 ceph-mon[29228]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:48:41.999 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:41 smithi105 ceph-mon[29228]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:41.999 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:41 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-06T02:48:41.999 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:41 smithi105 ceph-mon[29228]: osdmap e42: 6 total, 6 up, 6 in 2024-04-06T02:48:41.999 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:41 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-06T02:48:41.999 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:48:41.708+0000 7fd046604700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T02:48:42.369 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs dump' 2024-04-06T02:48:42.579 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:48:42.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:42 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-06T02:48:42.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:42 smithi184 ceph-mon[37439]: osdmap e43: 6 total, 6 up, 6 in 2024-04-06T02:48:42.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:42 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-06T02:48:42.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:42 smithi184 ceph-mon[37439]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T02:48:42.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:42 smithi184 ceph-mon[37439]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-06T02:48:42.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:42 smithi184 ceph-mon[37439]: osdmap e44: 6 total, 6 up, 6 in 2024-04-06T02:48:42.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:42 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-06T02:48:42.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:42 smithi184 ceph-mon[37439]: fsmap cephfs:0 2024-04-06T02:48:42.982 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:42 smithi184 ceph-mon[37439]: Saving service mds.cephfs spec with placement count:4 2024-04-06T02:48:42.982 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:42 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:42.982 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:42 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:48:42.982 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:42 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:48:42.982 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:42 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:48:42.982 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:42 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:42.982 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:42 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi105.mrivth", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T02:48:42.982 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:42 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi105.mrivth", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-06T02:48:42.982 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:42 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:48:43.017 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:42 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-06T02:48:43.017 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:42 smithi105 ceph-mon[29228]: osdmap e43: 6 total, 6 up, 6 in 2024-04-06T02:48:43.017 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:42 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-06T02:48:43.017 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:42 smithi105 ceph-mon[29228]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-06T02:48:43.017 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:42 smithi105 ceph-mon[29228]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-06T02:48:43.018 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:42 smithi105 ceph-mon[29228]: osdmap e44: 6 total, 6 up, 6 in 2024-04-06T02:48:43.018 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:42 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-06T02:48:43.018 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:42 smithi105 ceph-mon[29228]: fsmap cephfs:0 2024-04-06T02:48:43.018 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:42 smithi105 ceph-mon[29228]: Saving service mds.cephfs spec with placement count:4 2024-04-06T02:48:43.018 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:42 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:43.018 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:42 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:48:43.018 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:42 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:48:43.018 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:42 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:48:43.018 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:42 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:43.019 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:42 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi105.mrivth", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T02:48:43.019 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:42 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi105.mrivth", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-06T02:48:43.019 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:42 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:48:43.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:43 smithi105 ceph-mon[29228]: Deploying daemon mds.cephfs.smithi105.mrivth on smithi105 2024-04-06T02:48:43.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:43 smithi105 ceph-mon[29228]: pgmap v176: 65 pgs: 14 creating+peering, 13 active+clean, 38 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:43.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:43 smithi105 ceph-mon[29228]: osdmap e45: 6 total, 6 up, 6 in 2024-04-06T02:48:43.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:43 smithi184 ceph-mon[37439]: Deploying daemon mds.cephfs.smithi105.mrivth on smithi105 2024-04-06T02:48:43.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:43 smithi184 ceph-mon[37439]: pgmap v176: 65 pgs: 14 creating+peering, 13 active+clean, 38 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:43.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:43 smithi184 ceph-mon[37439]: osdmap e45: 6 total, 6 up, 6 in 2024-04-06T02:48:44.752 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:44 smithi105 ceph-mon[29228]: osdmap e46: 6 total, 6 up, 6 in 2024-04-06T02:48:44.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.867+0000 7ff0fcf27700 1 -- 172.21.15.105:0/921834330 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0f80ffac0 msgr2=0x7ff0f80ffee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:44.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.867+0000 7ff0fcf27700 1 --2- 172.21.15.105:0/921834330 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0f80ffac0 0x7ff0f80ffee0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec009910 tx=0x7ff0ec009c20 comp rx=0 tx=0).stop 2024-04-06T02:48:44.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.867+0000 7ff0fcf27700 1 -- 172.21.15.105:0/921834330 shutdown_connections 2024-04-06T02:48:44.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.867+0000 7ff0fcf27700 1 --2- 172.21.15.105:0/921834330 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0f8100cc0 0x7ff0f8101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:44.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.867+0000 7ff0fcf27700 1 --2- 172.21.15.105:0/921834330 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0f80ffac0 0x7ff0f80ffee0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:44.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.867+0000 7ff0fcf27700 1 -- 172.21.15.105:0/921834330 >> 172.21.15.105:0/921834330 conn(0x7ff0f80fb060 msgr2=0x7ff0f80fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:44.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.867+0000 7ff0fcf27700 1 -- 172.21.15.105:0/921834330 shutdown_connections 2024-04-06T02:48:44.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.867+0000 7ff0fcf27700 1 -- 172.21.15.105:0/921834330 wait complete. 2024-04-06T02:48:44.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.868+0000 7ff0fcf27700 1 Processor -- start 2024-04-06T02:48:44.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.868+0000 7ff0fcf27700 1 -- start start 2024-04-06T02:48:44.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.868+0000 7ff0fcf27700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0f80ffac0 0x7ff0f8196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:44.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.868+0000 7ff0fcf27700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0f8100cc0 0x7ff0f8196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:44.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.868+0000 7ff0fcf27700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f8196e00 con 0x7ff0f80ffac0 2024-04-06T02:48:44.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.868+0000 7ff0fcf27700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f8196f40 con 0x7ff0f8100cc0 2024-04-06T02:48:44.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.868+0000 7ff0f659c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0f80ffac0 0x7ff0f8196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:44.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.868+0000 7ff0f659c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0f80ffac0 0x7ff0f8196330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:52696/0 (socket says 172.21.15.105:52696) 2024-04-06T02:48:44.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.868+0000 7ff0f659c700 1 -- 172.21.15.105:0/320392807 learned_addr learned my addr 172.21.15.105:0/320392807 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:48:44.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.868+0000 7ff0f5d9b700 1 --2- 172.21.15.105:0/320392807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0f8100cc0 0x7ff0f8196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:44.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.868+0000 7ff0f659c700 1 -- 172.21.15.105:0/320392807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0f8100cc0 msgr2=0x7ff0f8196870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:44.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.869+0000 7ff0f659c700 1 --2- 172.21.15.105:0/320392807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0f8100cc0 0x7ff0f8196870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:44.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.869+0000 7ff0f659c700 1 -- 172.21.15.105:0/320392807 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ec0095f0 con 0x7ff0f80ffac0 2024-04-06T02:48:44.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.869+0000 7ff0f5d9b700 1 --2- 172.21.15.105:0/320392807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0f8100cc0 0x7ff0f8196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T02:48:44.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.869+0000 7ff0f659c700 1 --2- 172.21.15.105:0/320392807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0f80ffac0 0x7ff0f8196330 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec00bda0 tx=0x7ff0ec00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:44.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.869+0000 7ff0e77fe700 1 -- 172.21.15.105:0/320392807 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0ec01d070 con 0x7ff0f80ffac0 2024-04-06T02:48:44.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.869+0000 7ff0fcf27700 1 -- 172.21.15.105:0/320392807 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f819b9a0 con 0x7ff0f80ffac0 2024-04-06T02:48:44.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.869+0000 7ff0e77fe700 1 -- 172.21.15.105:0/320392807 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff0ec004c50 con 0x7ff0f80ffac0 2024-04-06T02:48:44.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.869+0000 7ff0e77fe700 1 -- 172.21.15.105:0/320392807 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0ec00f690 con 0x7ff0f80ffac0 2024-04-06T02:48:44.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.869+0000 7ff0fcf27700 1 -- 172.21.15.105:0/320392807 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f819be60 con 0x7ff0f80ffac0 2024-04-06T02:48:44.858 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.870+0000 7ff0fcf27700 1 -- 172.21.15.105:0/320392807 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0f8108620 con 0x7ff0f80ffac0 2024-04-06T02:48:44.858 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.870+0000 7ff0e77fe700 1 -- 172.21.15.105:0/320392807 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7ff0ec022a20 con 0x7ff0f80ffac0 2024-04-06T02:48:44.858 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.871+0000 7ff0e77fe700 1 --2- 172.21.15.105:0/320392807 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7ff0e006c4e0 0x7ff0e006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:44.858 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.871+0000 7ff0e77fe700 1 -- 172.21.15.105:0/320392807 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7ff0ec08cf40 con 0x7ff0f80ffac0 2024-04-06T02:48:44.858 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.871+0000 7ff0f5d9b700 1 --2- 172.21.15.105:0/320392807 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7ff0e006c4e0 0x7ff0e006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:44.858 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.871+0000 7ff0f5d9b700 1 --2- 172.21.15.105:0/320392807 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7ff0e006c4e0 0x7ff0e006e9a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8005fd0 tx=0x7ff0e8005e30 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:44.861 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:44.876+0000 7ff0e77fe700 1 -- 172.21.15.105:0/320392807 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff0ec0c00e0 con 0x7ff0f80ffac0 2024-04-06T02:48:44.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:44 smithi184 ceph-mon[37439]: osdmap e46: 6 total, 6 up, 6 in 2024-04-06T02:48:45.096 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:45.111+0000 7ff0fcf27700 1 -- 172.21.15.105:0/320392807 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff0f819c8b0 con 0x7ff0f80ffac0 2024-04-06T02:48:45.097 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:45.111+0000 7ff0e77fe700 1 -- 172.21.15.105:0/320392807 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1093 (secure 0 0 0) 0x7ff0ec05b260 con 0x7ff0f80ffac0 2024-04-06T02:48:45.097 INFO:teuthology.orchestra.run.smithi105.stdout:e2 2024-04-06T02:48:45.097 INFO:teuthology.orchestra.run.smithi105.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T02:48:45.097 INFO:teuthology.orchestra.run.smithi105.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T02:48:45.097 INFO:teuthology.orchestra.run.smithi105.stdout:legacy client fscid: 1 2024-04-06T02:48:45.097 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:48:45.097 INFO:teuthology.orchestra.run.smithi105.stdout:Filesystem 'cephfs' (1) 2024-04-06T02:48:45.098 INFO:teuthology.orchestra.run.smithi105.stdout:fs_name cephfs 2024-04-06T02:48:45.098 INFO:teuthology.orchestra.run.smithi105.stdout:epoch 2 2024-04-06T02:48:45.098 INFO:teuthology.orchestra.run.smithi105.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-06T02:48:45.098 INFO:teuthology.orchestra.run.smithi105.stdout:created 2024-04-06T02:48:41.709017+0000 2024-04-06T02:48:45.098 INFO:teuthology.orchestra.run.smithi105.stdout:modified 2024-04-06T02:48:41.709105+0000 2024-04-06T02:48:45.098 INFO:teuthology.orchestra.run.smithi105.stdout:tableserver 0 2024-04-06T02:48:45.098 INFO:teuthology.orchestra.run.smithi105.stdout:root 0 2024-04-06T02:48:45.098 INFO:teuthology.orchestra.run.smithi105.stdout:session_timeout 60 2024-04-06T02:48:45.098 INFO:teuthology.orchestra.run.smithi105.stdout:session_autoclose 300 2024-04-06T02:48:45.098 INFO:teuthology.orchestra.run.smithi105.stdout:max_file_size 1099511627776 2024-04-06T02:48:45.098 INFO:teuthology.orchestra.run.smithi105.stdout:required_client_features {} 2024-04-06T02:48:45.099 INFO:teuthology.orchestra.run.smithi105.stdout:last_failure 0 2024-04-06T02:48:45.099 INFO:teuthology.orchestra.run.smithi105.stdout:last_failure_osd_epoch 0 2024-04-06T02:48:45.099 INFO:teuthology.orchestra.run.smithi105.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T02:48:45.099 INFO:teuthology.orchestra.run.smithi105.stdout:max_mds 1 2024-04-06T02:48:45.099 INFO:teuthology.orchestra.run.smithi105.stdout:in 2024-04-06T02:48:45.099 INFO:teuthology.orchestra.run.smithi105.stdout:up {} 2024-04-06T02:48:45.099 INFO:teuthology.orchestra.run.smithi105.stdout:failed 2024-04-06T02:48:45.099 INFO:teuthology.orchestra.run.smithi105.stdout:damaged 2024-04-06T02:48:45.099 INFO:teuthology.orchestra.run.smithi105.stdout:stopped 2024-04-06T02:48:45.099 INFO:teuthology.orchestra.run.smithi105.stdout:data_pools [3] 2024-04-06T02:48:45.099 INFO:teuthology.orchestra.run.smithi105.stdout:metadata_pool 2 2024-04-06T02:48:45.099 INFO:teuthology.orchestra.run.smithi105.stdout:inline_data disabled 2024-04-06T02:48:45.099 INFO:teuthology.orchestra.run.smithi105.stdout:balancer 2024-04-06T02:48:45.100 INFO:teuthology.orchestra.run.smithi105.stdout:bal_rank_mask -1 2024-04-06T02:48:45.100 INFO:teuthology.orchestra.run.smithi105.stdout:standby_count_wanted 0 2024-04-06T02:48:45.100 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:48:45.100 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:48:45.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:45.113+0000 7ff0fcf27700 1 -- 172.21.15.105:0/320392807 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7ff0e006c4e0 msgr2=0x7ff0e006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:45.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:45.113+0000 7ff0fcf27700 1 --2- 172.21.15.105:0/320392807 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7ff0e006c4e0 0x7ff0e006e9a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8005fd0 tx=0x7ff0e8005e30 comp rx=0 tx=0).stop 2024-04-06T02:48:45.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:45.113+0000 7ff0fcf27700 1 -- 172.21.15.105:0/320392807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0f80ffac0 msgr2=0x7ff0f8196330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:45.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:45.113+0000 7ff0fcf27700 1 --2- 172.21.15.105:0/320392807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0f80ffac0 0x7ff0f8196330 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec00bda0 tx=0x7ff0ec00be80 comp rx=0 tx=0).stop 2024-04-06T02:48:45.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:45.113+0000 7ff0fcf27700 1 -- 172.21.15.105:0/320392807 shutdown_connections 2024-04-06T02:48:45.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:45.113+0000 7ff0fcf27700 1 --2- 172.21.15.105:0/320392807 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7ff0e006c4e0 0x7ff0e006e9a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:45.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:45.113+0000 7ff0fcf27700 1 --2- 172.21.15.105:0/320392807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0f80ffac0 0x7ff0f8196330 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:45.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:45.113+0000 7ff0fcf27700 1 --2- 172.21.15.105:0/320392807 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0f8100cc0 0x7ff0f8196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:45.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:45.113+0000 7ff0fcf27700 1 -- 172.21.15.105:0/320392807 >> 172.21.15.105:0/320392807 conn(0x7ff0f80fb060 msgr2=0x7ff0f8068db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:45.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:45.113+0000 7ff0fcf27700 1 -- 172.21.15.105:0/320392807 shutdown_connections 2024-04-06T02:48:45.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:45.113+0000 7ff0fcf27700 1 -- 172.21.15.105:0/320392807 wait complete. 2024-04-06T02:48:45.102 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 2 2024-04-06T02:48:45.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:45 smithi184 ceph-mon[37439]: pgmap v179: 65 pgs: 14 creating+peering, 13 active+clean, 38 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:45.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:45 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/320392807' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T02:48:46.003 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:45 smithi105 ceph-mon[29228]: pgmap v179: 65 pgs: 14 creating+peering, 13 active+clean, 38 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:46.003 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:45 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/320392807' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T02:48:46.349 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-06T02:48:46.363 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi105.front.sepia.ceph.com 2024-04-06T02:48:46.363 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs set cephfs max_mds 2' 2024-04-06T02:48:46.965 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:48:47.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:47 smithi184 ceph-mon[37439]: pgmap v180: 65 pgs: 9 creating+peering, 56 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:47.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:47 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:47.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:47 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:47.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:47 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:47.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:47 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi184.oochxp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T02:48:47.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:47 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi184.oochxp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-06T02:48:47.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:47 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:48:47.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:47 smithi184 ceph-mon[37439]: daemon mds.cephfs.smithi105.mrivth assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-06T02:48:47.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:47 smithi184 ceph-mon[37439]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-06T02:48:47.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:47 smithi184 ceph-mon[37439]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-06T02:48:47.732 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:47 smithi184 ceph-mon[37439]: Cluster is now healthy 2024-04-06T02:48:47.732 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:47 smithi184 ceph-mon[37439]: mds.? [v2:172.21.15.105:6826/2482683361,v1:172.21.15.105:6827/2482683361] up:boot 2024-04-06T02:48:47.732 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:47 smithi184 ceph-mon[37439]: fsmap cephfs:1 {0=cephfs.smithi105.mrivth=up:creating} 2024-04-06T02:48:47.732 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:47 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi105.mrivth"}]: dispatch 2024-04-06T02:48:47.732 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:47 smithi184 ceph-mon[37439]: daemon mds.cephfs.smithi105.mrivth is now active in filesystem cephfs as rank 0 2024-04-06T02:48:48.073 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:47 smithi105 ceph-mon[29228]: pgmap v180: 65 pgs: 9 creating+peering, 56 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:48.074 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:47 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:48.074 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:47 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:48.074 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:47 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:48.074 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:47 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi184.oochxp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T02:48:48.074 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:47 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi184.oochxp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-06T02:48:48.074 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:47 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:48:48.074 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:47 smithi105 ceph-mon[29228]: daemon mds.cephfs.smithi105.mrivth assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-06T02:48:48.074 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:47 smithi105 ceph-mon[29228]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-06T02:48:48.074 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:47 smithi105 ceph-mon[29228]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-06T02:48:48.074 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:47 smithi105 ceph-mon[29228]: Cluster is now healthy 2024-04-06T02:48:48.074 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:47 smithi105 ceph-mon[29228]: mds.? [v2:172.21.15.105:6826/2482683361,v1:172.21.15.105:6827/2482683361] up:boot 2024-04-06T02:48:48.075 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:47 smithi105 ceph-mon[29228]: fsmap cephfs:1 {0=cephfs.smithi105.mrivth=up:creating} 2024-04-06T02:48:48.075 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:47 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi105.mrivth"}]: dispatch 2024-04-06T02:48:48.075 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:47 smithi105 ceph-mon[29228]: daemon mds.cephfs.smithi105.mrivth is now active in filesystem cephfs as rank 0 2024-04-06T02:48:48.581 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.593+0000 7fcc9043e700 1 -- 172.21.15.105:0/2707505670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc880ffab0 msgr2=0x7fcc880ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:48.581 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.593+0000 7fcc9043e700 1 --2- 172.21.15.105:0/2707505670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc880ffab0 0x7fcc880ffed0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcc80009910 tx=0x7fcc80009c20 comp rx=0 tx=0).stop 2024-04-06T02:48:48.582 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.593+0000 7fcc9043e700 1 -- 172.21.15.105:0/2707505670 shutdown_connections 2024-04-06T02:48:48.582 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.593+0000 7fcc9043e700 1 --2- 172.21.15.105:0/2707505670 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc88100cb0 0x7fcc88101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:48.582 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.593+0000 7fcc9043e700 1 --2- 172.21.15.105:0/2707505670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc880ffab0 0x7fcc880ffed0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:48.582 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.593+0000 7fcc9043e700 1 -- 172.21.15.105:0/2707505670 >> 172.21.15.105:0/2707505670 conn(0x7fcc880fb030 msgr2=0x7fcc880fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:48.582 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.593+0000 7fcc9043e700 1 -- 172.21.15.105:0/2707505670 shutdown_connections 2024-04-06T02:48:48.582 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.593+0000 7fcc9043e700 1 -- 172.21.15.105:0/2707505670 wait complete. 2024-04-06T02:48:48.582 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.594+0000 7fcc9043e700 1 Processor -- start 2024-04-06T02:48:48.582 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.594+0000 7fcc9043e700 1 -- start start 2024-04-06T02:48:48.582 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.594+0000 7fcc9043e700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc880ffab0 0x7fcc88198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:48.582 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.594+0000 7fcc9043e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc88100cb0 0x7fcc88198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:48.582 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.594+0000 7fcc9043e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc88199090 con 0x7fcc88100cb0 2024-04-06T02:48:48.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.594+0000 7fcc9043e700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc881991d0 con 0x7fcc880ffab0 2024-04-06T02:48:48.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.594+0000 7fcc8d9d9700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc88100cb0 0x7fcc88198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:48.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.594+0000 7fcc8d9d9700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc88100cb0 0x7fcc88198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:52732/0 (socket says 172.21.15.105:52732) 2024-04-06T02:48:48.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.594+0000 7fcc8d9d9700 1 -- 172.21.15.105:0/1318595054 learned_addr learned my addr 172.21.15.105:0/1318595054 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:48:48.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.594+0000 7fcc8e1da700 1 --2- 172.21.15.105:0/1318595054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc880ffab0 0x7fcc88198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:48.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.595+0000 7fcc8d9d9700 1 -- 172.21.15.105:0/1318595054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc880ffab0 msgr2=0x7fcc88198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:48.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.595+0000 7fcc8d9d9700 1 --2- 172.21.15.105:0/1318595054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc880ffab0 0x7fcc88198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:48.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.595+0000 7fcc8d9d9700 1 -- 172.21.15.105:0/1318595054 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc800095f0 con 0x7fcc88100cb0 2024-04-06T02:48:48.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.595+0000 7fcc8e1da700 1 --2- 172.21.15.105:0/1318595054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc880ffab0 0x7fcc88198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T02:48:48.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.595+0000 7fcc8d9d9700 1 --2- 172.21.15.105:0/1318595054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc88100cb0 0x7fcc88198a70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcc7800c8a0 tx=0x7fcc7800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:48.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.595+0000 7fcc7f7fe700 1 -- 172.21.15.105:0/1318595054 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc7800cea0 con 0x7fcc88100cb0 2024-04-06T02:48:48.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.595+0000 7fcc7f7fe700 1 -- 172.21.15.105:0/1318595054 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcc78004830 con 0x7fcc88100cb0 2024-04-06T02:48:48.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.595+0000 7fcc7f7fe700 1 -- 172.21.15.105:0/1318595054 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc780055b0 con 0x7fcc88100cb0 2024-04-06T02:48:48.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.595+0000 7fcc9043e700 1 -- 172.21.15.105:0/1318595054 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc8819dc80 con 0x7fcc88100cb0 2024-04-06T02:48:48.584 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.595+0000 7fcc9043e700 1 -- 172.21.15.105:0/1318595054 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc8819e1d0 con 0x7fcc88100cb0 2024-04-06T02:48:48.584 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.596+0000 7fcc9043e700 1 -- 172.21.15.105:0/1318595054 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc88108610 con 0x7fcc88100cb0 2024-04-06T02:48:48.584 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.596+0000 7fcc7f7fe700 1 -- 172.21.15.105:0/1318595054 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fcc78007ad0 con 0x7fcc88100cb0 2024-04-06T02:48:48.584 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.597+0000 7fcc7f7fe700 1 --2- 172.21.15.105:0/1318595054 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fcc7406c4e0 0x7fcc7406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:48.584 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.597+0000 7fcc8e1da700 1 --2- 172.21.15.105:0/1318595054 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fcc7406c4e0 0x7fcc7406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:48.585 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.597+0000 7fcc7f7fe700 1 -- 172.21.15.105:0/1318595054 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fcc7808b480 con 0x7fcc88100cb0 2024-04-06T02:48:48.585 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.597+0000 7fcc8e1da700 1 --2- 172.21.15.105:0/1318595054 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fcc7406c4e0 0x7fcc7406e9a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcc80006010 tx=0x7fcc80005920 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:48.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.601+0000 7fcc7f7fe700 1 -- 172.21.15.105:0/1318595054 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fcc78059770 con 0x7fcc88100cb0 2024-04-06T02:48:48.790 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:48 smithi184 ceph-mon[37439]: Deploying daemon mds.cephfs.smithi184.oochxp on smithi184 2024-04-06T02:48:48.791 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:48 smithi184 ceph-mon[37439]: mds.? [v2:172.21.15.105:6826/2482683361,v1:172.21.15.105:6827/2482683361] up:active 2024-04-06T02:48:48.791 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:48 smithi184 ceph-mon[37439]: fsmap cephfs:1 {0=cephfs.smithi105.mrivth=up:active} 2024-04-06T02:48:48.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:48.839+0000 7fcc9043e700 1 -- 172.21.15.105:0/1318595054 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7fcc8819e890 con 0x7fcc88100cb0 2024-04-06T02:48:49.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:48 smithi105 ceph-mon[29228]: Deploying daemon mds.cephfs.smithi184.oochxp on smithi184 2024-04-06T02:48:49.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:48 smithi105 ceph-mon[29228]: mds.? [v2:172.21.15.105:6826/2482683361,v1:172.21.15.105:6827/2482683361] up:active 2024-04-06T02:48:49.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:48 smithi105 ceph-mon[29228]: fsmap cephfs:1 {0=cephfs.smithi105.mrivth=up:active} 2024-04-06T02:48:49.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:49.741+0000 7fcc7f7fe700 1 -- 172.21.15.105:0/1318595054 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v5) v1 ==== 105+0+0 (secure 0 0 0) 0x7fcc78059300 con 0x7fcc88100cb0 2024-04-06T02:48:49.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:49.745+0000 7fcc9043e700 1 -- 172.21.15.105:0/1318595054 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fcc7406c4e0 msgr2=0x7fcc7406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:49.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:49.745+0000 7fcc9043e700 1 --2- 172.21.15.105:0/1318595054 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fcc7406c4e0 0x7fcc7406e9a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcc80006010 tx=0x7fcc80005920 comp rx=0 tx=0).stop 2024-04-06T02:48:49.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:49.745+0000 7fcc9043e700 1 -- 172.21.15.105:0/1318595054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc88100cb0 msgr2=0x7fcc88198a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:49.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:49.745+0000 7fcc9043e700 1 --2- 172.21.15.105:0/1318595054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc88100cb0 0x7fcc88198a70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcc7800c8a0 tx=0x7fcc7800cc60 comp rx=0 tx=0).stop 2024-04-06T02:48:49.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:49.745+0000 7fcc9043e700 1 -- 172.21.15.105:0/1318595054 shutdown_connections 2024-04-06T02:48:49.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:49.745+0000 7fcc9043e700 1 --2- 172.21.15.105:0/1318595054 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fcc7406c4e0 0x7fcc7406e9a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:49.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:49.745+0000 7fcc9043e700 1 --2- 172.21.15.105:0/1318595054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc880ffab0 0x7fcc88198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:49.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:49.745+0000 7fcc9043e700 1 --2- 172.21.15.105:0/1318595054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc88100cb0 0x7fcc88198a70 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:49.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:49.745+0000 7fcc9043e700 1 -- 172.21.15.105:0/1318595054 >> 172.21.15.105:0/1318595054 conn(0x7fcc880fb030 msgr2=0x7fcc880683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:49.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:49.745+0000 7fcc9043e700 1 -- 172.21.15.105:0/1318595054 shutdown_connections 2024-04-06T02:48:49.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:49.745+0000 7fcc9043e700 1 -- 172.21.15.105:0/1318595054 wait complete. 2024-04-06T02:48:49.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:49 smithi184 ceph-mon[37439]: pgmap v181: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:49.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:49 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/1318595054' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-04-06T02:48:49.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:49 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:49.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:49 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:49.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:49 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:49.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:49 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi105.oxxcbq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T02:48:49.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:49 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi105.oxxcbq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-06T02:48:49.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:49 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:48:50.046 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:49 smithi105 ceph-mon[29228]: pgmap v181: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:48:50.047 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:49 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/1318595054' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-04-06T02:48:50.047 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:49 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:50.047 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:49 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:50.047 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:49 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:50.047 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:49 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi105.oxxcbq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T02:48:50.047 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:49 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi105.oxxcbq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-06T02:48:50.047 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:49 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:48:50.376 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-06T02:48:50.386 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi105.front.sepia.ceph.com 2024-04-06T02:48:50.386 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2024-04-06T02:48:50.859 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:50 smithi184 ceph-mon[37439]: Deploying daemon mds.cephfs.smithi105.oxxcbq on smithi105 2024-04-06T02:48:50.859 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:50 smithi184 ceph-mon[37439]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-06T02:48:50.859 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:50 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/1318595054' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-04-06T02:48:50.859 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:50 smithi184 ceph-mon[37439]: mds.? [v2:172.21.15.184:6824/2770403500,v1:172.21.15.184:6825/2770403500] up:boot 2024-04-06T02:48:50.859 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:50 smithi184 ceph-mon[37439]: daemon mds.cephfs.smithi184.oochxp assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-06T02:48:50.859 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:50 smithi184 ceph-mon[37439]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-06T02:48:50.859 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:50 smithi184 ceph-mon[37439]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-06T02:48:50.859 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:50 smithi184 ceph-mon[37439]: fsmap cephfs:1 {0=cephfs.smithi105.mrivth=up:active} 1 up:standby 2024-04-06T02:48:50.859 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:50 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi184.oochxp"}]: dispatch 2024-04-06T02:48:50.859 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:50 smithi184 ceph-mon[37439]: fsmap cephfs:2 {0=cephfs.smithi105.mrivth=up:active,1=cephfs.smithi184.oochxp=up:creating} 2024-04-06T02:48:50.859 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:50 smithi184 ceph-mon[37439]: daemon mds.cephfs.smithi184.oochxp is now active in filesystem cephfs as rank 1 2024-04-06T02:48:51.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:50 smithi105 ceph-mon[29228]: Deploying daemon mds.cephfs.smithi105.oxxcbq on smithi105 2024-04-06T02:48:51.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:50 smithi105 ceph-mon[29228]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-06T02:48:51.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:50 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/1318595054' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-04-06T02:48:51.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:50 smithi105 ceph-mon[29228]: mds.? [v2:172.21.15.184:6824/2770403500,v1:172.21.15.184:6825/2770403500] up:boot 2024-04-06T02:48:51.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:50 smithi105 ceph-mon[29228]: daemon mds.cephfs.smithi184.oochxp assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-06T02:48:51.176 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:50 smithi105 ceph-mon[29228]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-06T02:48:51.176 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:50 smithi105 ceph-mon[29228]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-06T02:48:51.176 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:50 smithi105 ceph-mon[29228]: fsmap cephfs:1 {0=cephfs.smithi105.mrivth=up:active} 1 up:standby 2024-04-06T02:48:51.176 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:50 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi184.oochxp"}]: dispatch 2024-04-06T02:48:51.176 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:50 smithi105 ceph-mon[29228]: fsmap cephfs:2 {0=cephfs.smithi105.mrivth=up:active,1=cephfs.smithi184.oochxp=up:creating} 2024-04-06T02:48:51.176 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:50 smithi105 ceph-mon[29228]: daemon mds.cephfs.smithi184.oochxp is now active in filesystem cephfs as rank 1 2024-04-06T02:48:51.675 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:48:51.797 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:51 smithi105 ceph-mon[29228]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s wr, 3 op/s 2024-04-06T02:48:51.797 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:51 smithi105 ceph-mon[29228]: mds.? [v2:172.21.15.184:6824/2770403500,v1:172.21.15.184:6825/2770403500] up:active 2024-04-06T02:48:51.797 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:51 smithi105 ceph-mon[29228]: fsmap cephfs:2 {0=cephfs.smithi105.mrivth=up:active,1=cephfs.smithi184.oochxp=up:active} 2024-04-06T02:48:52.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:51 smithi184 ceph-mon[37439]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s wr, 3 op/s 2024-04-06T02:48:52.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:51 smithi184 ceph-mon[37439]: mds.? [v2:172.21.15.184:6824/2770403500,v1:172.21.15.184:6825/2770403500] up:active 2024-04-06T02:48:52.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:51 smithi184 ceph-mon[37439]: fsmap cephfs:2 {0=cephfs.smithi105.mrivth=up:active,1=cephfs.smithi184.oochxp=up:active} 2024-04-06T02:48:53.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.417+0000 7fcad62f4700 1 -- 172.21.15.105:0/2558758824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcad0100cb0 msgr2=0x7fcad0101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:53.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.417+0000 7fcad62f4700 1 --2- 172.21.15.105:0/2558758824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcad0100cb0 0x7fcad0101110 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fcac4009910 tx=0x7fcac4009c20 comp rx=0 tx=0).stop 2024-04-06T02:48:53.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.417+0000 7fcad62f4700 1 -- 172.21.15.105:0/2558758824 shutdown_connections 2024-04-06T02:48:53.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.417+0000 7fcad62f4700 1 --2- 172.21.15.105:0/2558758824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcad0100cb0 0x7fcad0101110 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:53.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.417+0000 7fcad62f4700 1 --2- 172.21.15.105:0/2558758824 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcad00ffab0 0x7fcad00ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:53.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.417+0000 7fcad62f4700 1 -- 172.21.15.105:0/2558758824 >> 172.21.15.105:0/2558758824 conn(0x7fcad00fb030 msgr2=0x7fcad00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:53.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.417+0000 7fcad62f4700 1 -- 172.21.15.105:0/2558758824 shutdown_connections 2024-04-06T02:48:53.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.417+0000 7fcad62f4700 1 -- 172.21.15.105:0/2558758824 wait complete. 2024-04-06T02:48:53.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.418+0000 7fcad62f4700 1 Processor -- start 2024-04-06T02:48:53.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.418+0000 7fcad62f4700 1 -- start start 2024-04-06T02:48:53.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.418+0000 7fcad62f4700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcad00ffab0 0x7fcad0068f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:53.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.418+0000 7fcad62f4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcad0100cb0 0x7fcad019b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:53.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.418+0000 7fcad62f4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad019b690 con 0x7fcad0100cb0 2024-04-06T02:48:53.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.418+0000 7fcad62f4700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad019b7d0 con 0x7fcad00ffab0 2024-04-06T02:48:53.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.418+0000 7fcacf7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcad0100cb0 0x7fcad019b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:53.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.418+0000 7fcacf7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcad0100cb0 0x7fcad019b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47286/0 (socket says 172.21.15.105:47286) 2024-04-06T02:48:53.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.418+0000 7fcacf7fe700 1 -- 172.21.15.105:0/2253633360 learned_addr learned my addr 172.21.15.105:0/2253633360 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:48:53.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.419+0000 7fcacffff700 1 --2- 172.21.15.105:0/2253633360 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcad00ffab0 0x7fcad0068f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:53.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.419+0000 7fcacf7fe700 1 -- 172.21.15.105:0/2253633360 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcad00ffab0 msgr2=0x7fcad0068f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:53.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.419+0000 7fcacf7fe700 1 --2- 172.21.15.105:0/2253633360 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcad00ffab0 0x7fcad0068f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:53.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.419+0000 7fcacf7fe700 1 -- 172.21.15.105:0/2253633360 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac40095f0 con 0x7fcad0100cb0 2024-04-06T02:48:53.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.419+0000 7fcacf7fe700 1 --2- 172.21.15.105:0/2253633360 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcad0100cb0 0x7fcad019b0c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fcac4006010 tx=0x7fcac400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:53.407 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.419+0000 7fcacd7fa700 1 -- 172.21.15.105:0/2253633360 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcac401d070 con 0x7fcad0100cb0 2024-04-06T02:48:53.407 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.419+0000 7fcacd7fa700 1 -- 172.21.15.105:0/2253633360 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcac400f460 con 0x7fcad0100cb0 2024-04-06T02:48:53.407 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.419+0000 7fcacd7fa700 1 -- 172.21.15.105:0/2253633360 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcac4017630 con 0x7fcad0100cb0 2024-04-06T02:48:53.407 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.419+0000 7fcad62f4700 1 -- 172.21.15.105:0/2253633360 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad019ba50 con 0x7fcad0100cb0 2024-04-06T02:48:53.407 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.419+0000 7fcad62f4700 1 -- 172.21.15.105:0/2253633360 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad019bf40 con 0x7fcad0100cb0 2024-04-06T02:48:53.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.420+0000 7fcad62f4700 1 -- 172.21.15.105:0/2253633360 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcad0108610 con 0x7fcad0100cb0 2024-04-06T02:48:53.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.420+0000 7fcacd7fa700 1 -- 172.21.15.105:0/2253633360 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fcac400f5d0 con 0x7fcad0100cb0 2024-04-06T02:48:53.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.421+0000 7fcacd7fa700 1 --2- 172.21.15.105:0/2253633360 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fcab806c4e0 0x7fcab806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:53.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.421+0000 7fcacd7fa700 1 -- 172.21.15.105:0/2253633360 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fcac408cdd0 con 0x7fcad0100cb0 2024-04-06T02:48:53.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.421+0000 7fcacffff700 1 --2- 172.21.15.105:0/2253633360 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fcab806c4e0 0x7fcab806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:53.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.421+0000 7fcacffff700 1 --2- 172.21.15.105:0/2253633360 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fcab806c4e0 0x7fcab806e9a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcac0005950 tx=0x7fcac00058e0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:53.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.428+0000 7fcacd7fa700 1 -- 172.21.15.105:0/2253633360 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fcac405b040 con 0x7fcad0100cb0 2024-04-06T02:48:53.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:53.668+0000 7fcad62f4700 1 -- 172.21.15.105:0/2253633360 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fcad0108830 con 0x7fcad0100cb0 2024-04-06T02:48:53.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:53 smithi184 ceph-mon[37439]: pgmap v183: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s wr, 6 op/s 2024-04-06T02:48:53.769 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:53 smithi105 ceph-mon[29228]: pgmap v183: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s wr, 6 op/s 2024-04-06T02:48:54.486 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:54.501+0000 7fcacd7fa700 1 -- 172.21.15.105:0/2253633360 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcac405abd0 con 0x7fcad0100cb0 2024-04-06T02:48:54.488 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:54.503+0000 7fcad62f4700 1 -- 172.21.15.105:0/2253633360 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fcab806c4e0 msgr2=0x7fcab806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:54.488 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:54.503+0000 7fcad62f4700 1 --2- 172.21.15.105:0/2253633360 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fcab806c4e0 0x7fcab806e9a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcac0005950 tx=0x7fcac00058e0 comp rx=0 tx=0).stop 2024-04-06T02:48:54.489 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:54.503+0000 7fcad62f4700 1 -- 172.21.15.105:0/2253633360 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcad0100cb0 msgr2=0x7fcad019b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:54.489 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:54.503+0000 7fcad62f4700 1 --2- 172.21.15.105:0/2253633360 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcad0100cb0 0x7fcad019b0c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fcac4006010 tx=0x7fcac400be20 comp rx=0 tx=0).stop 2024-04-06T02:48:54.489 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:54.503+0000 7fcad62f4700 1 -- 172.21.15.105:0/2253633360 shutdown_connections 2024-04-06T02:48:54.489 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:54.503+0000 7fcad62f4700 1 --2- 172.21.15.105:0/2253633360 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fcab806c4e0 0x7fcab806e9a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:54.489 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:54.503+0000 7fcad62f4700 1 --2- 172.21.15.105:0/2253633360 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcad00ffab0 0x7fcad0068f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:54.489 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:54.503+0000 7fcad62f4700 1 --2- 172.21.15.105:0/2253633360 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcad0100cb0 0x7fcad019b0c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:54.489 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:54.503+0000 7fcad62f4700 1 -- 172.21.15.105:0/2253633360 >> 172.21.15.105:0/2253633360 conn(0x7fcad00fb030 msgr2=0x7fcad0103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:54.489 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:54.503+0000 7fcad62f4700 1 -- 172.21.15.105:0/2253633360 shutdown_connections 2024-04-06T02:48:54.489 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:54.503+0000 7fcad62f4700 1 -- 172.21.15.105:0/2253633360 wait complete. 2024-04-06T02:48:54.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:54 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/2253633360' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-04-06T02:48:54.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:54 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:54.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:54 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:54.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:54 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:54.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:54 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi184.htgrjn", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T02:48:54.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:54 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi184.htgrjn", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-06T02:48:54.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:54 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:48:54.780 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:54 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/2253633360' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-04-06T02:48:54.780 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:54 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:54.780 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:54 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:54.781 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:54 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:54.781 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:54 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi184.htgrjn", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-06T02:48:54.781 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:54 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi184.htgrjn", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-06T02:48:54.781 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:54 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:48:55.320 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-06T02:48:55.330 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi105.front.sepia.ceph.com 2024-04-06T02:48:55.330 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-04-06T02:48:55.547 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:48:55.669 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:55 smithi105 ceph-mon[29228]: Deploying daemon mds.cephfs.smithi184.htgrjn on smithi184 2024-04-06T02:48:55.669 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:55 smithi105 ceph-mon[29228]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-06T02:48:55.669 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:55 smithi105 ceph-mon[29228]: Cluster is now healthy 2024-04-06T02:48:55.669 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:55 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/2253633360' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-04-06T02:48:55.669 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:55 smithi105 ceph-mon[29228]: mds.? [v2:172.21.15.105:6828/2512987070,v1:172.21.15.105:6829/2512987070] up:boot 2024-04-06T02:48:55.669 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:55 smithi105 ceph-mon[29228]: fsmap cephfs:2 {0=cephfs.smithi105.mrivth=up:active,1=cephfs.smithi184.oochxp=up:active} 1 up:standby 2024-04-06T02:48:55.669 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:55 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi105.oxxcbq"}]: dispatch 2024-04-06T02:48:55.669 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:55 smithi105 ceph-mon[29228]: pgmap v184: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s wr, 6 op/s 2024-04-06T02:48:55.669 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:55 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:48:55.670 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:55 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:55.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:55 smithi184 ceph-mon[37439]: Deploying daemon mds.cephfs.smithi184.htgrjn on smithi184 2024-04-06T02:48:55.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:55 smithi184 ceph-mon[37439]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-06T02:48:55.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:55 smithi184 ceph-mon[37439]: Cluster is now healthy 2024-04-06T02:48:55.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:55 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/2253633360' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-04-06T02:48:55.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:55 smithi184 ceph-mon[37439]: mds.? [v2:172.21.15.105:6828/2512987070,v1:172.21.15.105:6829/2512987070] up:boot 2024-04-06T02:48:55.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:55 smithi184 ceph-mon[37439]: fsmap cephfs:2 {0=cephfs.smithi105.mrivth=up:active,1=cephfs.smithi184.oochxp=up:active} 1 up:standby 2024-04-06T02:48:55.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:55 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi105.oxxcbq"}]: dispatch 2024-04-06T02:48:55.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:55 smithi184 ceph-mon[37439]: pgmap v184: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s wr, 6 op/s 2024-04-06T02:48:55.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:55 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:48:55.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:55 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:57.284 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.296+0000 7fc3c9735700 1 -- 172.21.15.105:0/238587125 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3c4101dc0 msgr2=0x7fc3c4102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:57.284 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.296+0000 7fc3c9735700 1 --2- 172.21.15.105:0/238587125 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3c4101dc0 0x7fc3c4102220 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac009910 tx=0x7fc3ac009c20 comp rx=0 tx=0).stop 2024-04-06T02:48:57.285 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.296+0000 7fc3c9735700 1 -- 172.21.15.105:0/238587125 shutdown_connections 2024-04-06T02:48:57.285 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.296+0000 7fc3c9735700 1 --2- 172.21.15.105:0/238587125 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3c4101dc0 0x7fc3c4102220 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:57.285 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.296+0000 7fc3c9735700 1 --2- 172.21.15.105:0/238587125 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3c4100bc0 0x7fc3c4100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:57.285 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.296+0000 7fc3c9735700 1 -- 172.21.15.105:0/238587125 >> 172.21.15.105:0/238587125 conn(0x7fc3c40fc140 msgr2=0x7fc3c40fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:57.285 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.296+0000 7fc3c9735700 1 -- 172.21.15.105:0/238587125 shutdown_connections 2024-04-06T02:48:57.285 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.296+0000 7fc3c9735700 1 -- 172.21.15.105:0/238587125 wait complete. 2024-04-06T02:48:57.285 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.297+0000 7fc3c9735700 1 Processor -- start 2024-04-06T02:48:57.285 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.297+0000 7fc3c9735700 1 -- start start 2024-04-06T02:48:57.285 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.297+0000 7fc3c9735700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3c4100bc0 0x7fc3c4196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:57.285 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.297+0000 7fc3c9735700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3c4101dc0 0x7fc3c41968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:57.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.297+0000 7fc3c9735700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c4196ed0 con 0x7fc3c4101dc0 2024-04-06T02:48:57.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.297+0000 7fc3c9735700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c4197010 con 0x7fc3c4100bc0 2024-04-06T02:48:57.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.297+0000 7fc3c27fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3c4101dc0 0x7fc3c41968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:57.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.297+0000 7fc3c27fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3c4101dc0 0x7fc3c41968b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47320/0 (socket says 172.21.15.105:47320) 2024-04-06T02:48:57.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.297+0000 7fc3c2ffd700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3c4100bc0 0x7fc3c4196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:57.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.297+0000 7fc3c27fc700 1 -- 172.21.15.105:0/3012548946 learned_addr learned my addr 172.21.15.105:0/3012548946 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:48:57.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.298+0000 7fc3c2ffd700 1 -- 172.21.15.105:0/3012548946 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3c4101dc0 msgr2=0x7fc3c41968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:57.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.298+0000 7fc3c2ffd700 1 --2- 172.21.15.105:0/3012548946 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3c4101dc0 0x7fc3c41968b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:57.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.298+0000 7fc3c2ffd700 1 -- 172.21.15.105:0/3012548946 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ac0095f0 con 0x7fc3c4100bc0 2024-04-06T02:48:57.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.298+0000 7fc3c2ffd700 1 --2- 172.21.15.105:0/3012548946 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3c4100bc0 0x7fc3c4196370 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc3b400e940 tx=0x7fc3b400ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:57.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.298+0000 7fc3bbfff700 1 -- 172.21.15.105:0/3012548946 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3b40041f0 con 0x7fc3c4100bc0 2024-04-06T02:48:57.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.298+0000 7fc3bbfff700 1 -- 172.21.15.105:0/3012548946 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc3b4004350 con 0x7fc3c4100bc0 2024-04-06T02:48:57.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.298+0000 7fc3bbfff700 1 -- 172.21.15.105:0/3012548946 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3b4003c80 con 0x7fc3c4100bc0 2024-04-06T02:48:57.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.298+0000 7fc3c9735700 1 -- 172.21.15.105:0/3012548946 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3c419bac0 con 0x7fc3c4100bc0 2024-04-06T02:48:57.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.298+0000 7fc3c9735700 1 -- 172.21.15.105:0/3012548946 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3c419c010 con 0x7fc3c4100bc0 2024-04-06T02:48:57.288 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.299+0000 7fc3c9735700 1 -- 172.21.15.105:0/3012548946 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3c4109720 con 0x7fc3c4100bc0 2024-04-06T02:48:57.288 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.300+0000 7fc3bbfff700 1 -- 172.21.15.105:0/3012548946 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fc3b400c7c0 con 0x7fc3c4100bc0 2024-04-06T02:48:57.288 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.300+0000 7fc3bbfff700 1 --2- 172.21.15.105:0/3012548946 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fc3b006c4e0 0x7fc3b006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:48:57.288 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.300+0000 7fc3bbfff700 1 -- 172.21.15.105:0/3012548946 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fc3b4014070 con 0x7fc3c4100bc0 2024-04-06T02:48:57.288 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.300+0000 7fc3c27fc700 1 --2- 172.21.15.105:0/3012548946 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fc3b006c4e0 0x7fc3b006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:48:57.288 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.301+0000 7fc3c27fc700 1 --2- 172.21.15.105:0/3012548946 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fc3b006c4e0 0x7fc3b006e9a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac009fd0 tx=0x7fc3ac005940 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:48:57.290 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.305+0000 7fc3bbfff700 1 -- 172.21.15.105:0/3012548946 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc3b408de60 con 0x7fc3c4100bc0 2024-04-06T02:48:57.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:57.561+0000 7fc3c9735700 1 -- 172.21.15.105:0/3012548946 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc3c41098e0 con 0x7fc3c4100bc0 2024-04-06T02:48:57.572 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:57 smithi184 ceph-mon[37439]: mds.? [v2:172.21.15.184:6826/323195824,v1:172.21.15.184:6827/323195824] up:boot 2024-04-06T02:48:57.572 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:57 smithi184 ceph-mon[37439]: fsmap cephfs:2 {0=cephfs.smithi105.mrivth=up:active,1=cephfs.smithi184.oochxp=up:active} 2 up:standby 2024-04-06T02:48:57.572 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:57 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi184.htgrjn"}]: dispatch 2024-04-06T02:48:57.572 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:57 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:57.572 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:57 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:57.573 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:57 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:57.573 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:57 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:57.573 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:57 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:57.573 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:57 smithi184 ceph-mon[37439]: pgmap v185: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s wr, 6 op/s 2024-04-06T02:48:57.573 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:57 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:48:57.760 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:57 smithi105 ceph-mon[29228]: mds.? [v2:172.21.15.184:6826/323195824,v1:172.21.15.184:6827/323195824] up:boot 2024-04-06T02:48:57.760 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:57 smithi105 ceph-mon[29228]: fsmap cephfs:2 {0=cephfs.smithi105.mrivth=up:active,1=cephfs.smithi184.oochxp=up:active} 2 up:standby 2024-04-06T02:48:57.760 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:57 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi184.htgrjn"}]: dispatch 2024-04-06T02:48:57.760 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:57 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:57.760 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:57 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:57.761 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:57 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:57.761 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:57 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:57.761 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:57 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:57.761 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:57 smithi105 ceph-mon[29228]: pgmap v185: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s wr, 6 op/s 2024-04-06T02:48:57.761 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:57 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:48:58.496 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:58.510+0000 7fc3bbfff700 1 -- 172.21.15.105:0/3012548946 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v10) v1 ==== 168+0+0 (secure 0 0 0) 0x7fc3b4059e40 con 0x7fc3c4100bc0 2024-04-06T02:48:58.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:58.513+0000 7fc3c9735700 1 -- 172.21.15.105:0/3012548946 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fc3b006c4e0 msgr2=0x7fc3b006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:58.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:58.513+0000 7fc3c9735700 1 --2- 172.21.15.105:0/3012548946 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fc3b006c4e0 0x7fc3b006e9a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac009fd0 tx=0x7fc3ac005940 comp rx=0 tx=0).stop 2024-04-06T02:48:58.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:58.513+0000 7fc3c9735700 1 -- 172.21.15.105:0/3012548946 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3c4100bc0 msgr2=0x7fc3c4196370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:48:58.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:58.513+0000 7fc3c9735700 1 --2- 172.21.15.105:0/3012548946 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3c4100bc0 0x7fc3c4196370 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc3b400e940 tx=0x7fc3b400ec50 comp rx=0 tx=0).stop 2024-04-06T02:48:58.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:58.513+0000 7fc3c9735700 1 -- 172.21.15.105:0/3012548946 shutdown_connections 2024-04-06T02:48:58.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:58.513+0000 7fc3c9735700 1 --2- 172.21.15.105:0/3012548946 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fc3b006c4e0 0x7fc3b006e9a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:58.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:58.513+0000 7fc3c9735700 1 --2- 172.21.15.105:0/3012548946 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc3c4100bc0 0x7fc3c4196370 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:58.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:58.513+0000 7fc3c9735700 1 --2- 172.21.15.105:0/3012548946 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3c4101dc0 0x7fc3c41968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:48:58.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:58.513+0000 7fc3c9735700 1 -- 172.21.15.105:0/3012548946 >> 172.21.15.105:0/3012548946 conn(0x7fc3c40fc140 msgr2=0x7fc3c4104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:48:58.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:58.513+0000 7fc3c9735700 1 -- 172.21.15.105:0/3012548946 shutdown_connections 2024-04-06T02:48:58.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:48:58.513+0000 7fc3c9735700 1 -- 172.21.15.105:0/3012548946 wait complete. 2024-04-06T02:48:58.500 INFO:teuthology.orchestra.run.smithi105.stderr:inline data enabled 2024-04-06T02:48:58.860 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:58 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/3012548946' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T02:48:58.860 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:58 smithi105 ceph-mon[29228]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T02:48:58.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:58 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/3012548946' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T02:48:58.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:58 smithi184 ceph-mon[37439]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T02:48:59.248 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-06T02:48:59.258 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi105.front.sepia.ceph.com 2024-04-06T02:48:59.258 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs dump' 2024-04-06T02:48:59.873 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:59 smithi105 ceph-mon[29228]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-04-06T02:48:59.874 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:59 smithi105 ceph-mon[29228]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-04-06T02:48:59.874 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:59 smithi105 ceph-mon[29228]: mds.? [v2:172.21.15.184:6824/2770403500,v1:172.21.15.184:6825/2770403500] up:active 2024-04-06T02:48:59.874 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:59 smithi105 ceph-mon[29228]: mds.? [v2:172.21.15.105:6828/2512987070,v1:172.21.15.105:6829/2512987070] up:standby 2024-04-06T02:48:59.874 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:59 smithi105 ceph-mon[29228]: Dropping low affinity active daemon mds.cephfs.smithi105.mrivth in favor of higher affinity standby. 2024-04-06T02:48:59.874 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:59 smithi105 ceph-mon[29228]: Replacing daemon mds.cephfs.smithi105.mrivth as rank 0 with standby daemon mds.cephfs.smithi105.oxxcbq 2024-04-06T02:48:59.874 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:59 smithi105 ceph-mon[29228]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-06T02:48:59.874 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:59 smithi105 ceph-mon[29228]: fsmap cephfs:2 {0=cephfs.smithi105.mrivth=up:active,1=cephfs.smithi184.oochxp=up:active} 2 up:standby 2024-04-06T02:48:59.874 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:59 smithi105 ceph-mon[29228]: osdmap e47: 6 total, 6 up, 6 in 2024-04-06T02:48:59.874 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:59 smithi105 ceph-mon[29228]: fsmap cephfs:2/2 {0=cephfs.smithi105.oxxcbq=up:replay,1=cephfs.smithi184.oochxp=up:active} 1 up:standby 2024-04-06T02:48:59.874 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:59 smithi105 ceph-mon[29228]: pgmap v187: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s wr, 8 op/s 2024-04-06T02:48:59.874 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:59 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:59.874 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:48:59 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:59.970 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:48:59.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:59 smithi184 ceph-mon[37439]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-04-06T02:48:59.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:59 smithi184 ceph-mon[37439]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-04-06T02:48:59.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:59 smithi184 ceph-mon[37439]: mds.? [v2:172.21.15.184:6824/2770403500,v1:172.21.15.184:6825/2770403500] up:active 2024-04-06T02:48:59.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:59 smithi184 ceph-mon[37439]: mds.? [v2:172.21.15.105:6828/2512987070,v1:172.21.15.105:6829/2512987070] up:standby 2024-04-06T02:48:59.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:59 smithi184 ceph-mon[37439]: Dropping low affinity active daemon mds.cephfs.smithi105.mrivth in favor of higher affinity standby. 2024-04-06T02:48:59.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:59 smithi184 ceph-mon[37439]: Replacing daemon mds.cephfs.smithi105.mrivth as rank 0 with standby daemon mds.cephfs.smithi105.oxxcbq 2024-04-06T02:48:59.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:59 smithi184 ceph-mon[37439]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-06T02:48:59.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:59 smithi184 ceph-mon[37439]: fsmap cephfs:2 {0=cephfs.smithi105.mrivth=up:active,1=cephfs.smithi184.oochxp=up:active} 2 up:standby 2024-04-06T02:48:59.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:59 smithi184 ceph-mon[37439]: osdmap e47: 6 total, 6 up, 6 in 2024-04-06T02:48:59.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:59 smithi184 ceph-mon[37439]: fsmap cephfs:2/2 {0=cephfs.smithi105.oxxcbq=up:replay,1=cephfs.smithi184.oochxp=up:active} 1 up:standby 2024-04-06T02:48:59.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:59 smithi184 ceph-mon[37439]: pgmap v187: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s wr, 8 op/s 2024-04-06T02:48:59.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:59 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:48:59.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:48:59 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:49:00.913 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:00 smithi105 ceph-mon[29228]: mds.? [v2:172.21.15.105:6828/2512987070,v1:172.21.15.105:6829/2512987070] up:resolve 2024-04-06T02:49:00.913 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:00 smithi105 ceph-mon[29228]: mds.? [v2:172.21.15.105:6826/2109838618,v1:172.21.15.105:6827/2109838618] up:boot 2024-04-06T02:49:00.913 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:00 smithi105 ceph-mon[29228]: fsmap cephfs:2/2 {0=cephfs.smithi105.oxxcbq=up:resolve,1=cephfs.smithi184.oochxp=up:active} 2 up:standby 2024-04-06T02:49:00.913 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:00 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi105.mrivth"}]: dispatch 2024-04-06T02:49:00.913 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:00 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:49:00.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:00 smithi184 ceph-mon[37439]: mds.? [v2:172.21.15.105:6828/2512987070,v1:172.21.15.105:6829/2512987070] up:resolve 2024-04-06T02:49:00.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:00 smithi184 ceph-mon[37439]: mds.? [v2:172.21.15.105:6826/2109838618,v1:172.21.15.105:6827/2109838618] up:boot 2024-04-06T02:49:00.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:00 smithi184 ceph-mon[37439]: fsmap cephfs:2/2 {0=cephfs.smithi105.oxxcbq=up:resolve,1=cephfs.smithi184.oochxp=up:active} 2 up:standby 2024-04-06T02:49:00.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:00 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi105.mrivth"}]: dispatch 2024-04-06T02:49:00.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:00 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:49:01.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:01 smithi105 ceph-mon[29228]: mds.? [v2:172.21.15.105:6828/2512987070,v1:172.21.15.105:6829/2512987070] up:reconnect 2024-04-06T02:49:01.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:01 smithi105 ceph-mon[29228]: mds.? [v2:172.21.15.184:6826/323195824,v1:172.21.15.184:6827/323195824] up:standby 2024-04-06T02:49:01.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:01 smithi105 ceph-mon[29228]: fsmap cephfs:2/2 {0=cephfs.smithi105.oxxcbq=up:reconnect,1=cephfs.smithi184.oochxp=up:active} 2 up:standby 2024-04-06T02:49:01.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:01 smithi105 ceph-mon[29228]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 819 B/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-06T02:49:01.859 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:01 smithi184 ceph-mon[37439]: mds.? [v2:172.21.15.105:6828/2512987070,v1:172.21.15.105:6829/2512987070] up:reconnect 2024-04-06T02:49:01.859 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:01 smithi184 ceph-mon[37439]: mds.? [v2:172.21.15.184:6826/323195824,v1:172.21.15.184:6827/323195824] up:standby 2024-04-06T02:49:01.859 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:01 smithi184 ceph-mon[37439]: fsmap cephfs:2/2 {0=cephfs.smithi105.oxxcbq=up:reconnect,1=cephfs.smithi184.oochxp=up:active} 2 up:standby 2024-04-06T02:49:01.859 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:01 smithi184 ceph-mon[37439]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 819 B/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-06T02:49:01.918 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.932+0000 7fda3f358700 1 -- 172.21.15.105:0/123150478 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda38100bc0 msgr2=0x7fda38100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:01.918 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.932+0000 7fda3f358700 1 --2- 172.21.15.105:0/123150478 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda38100bc0 0x7fda38100fe0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fda34009910 tx=0x7fda34009c20 comp rx=0 tx=0).stop 2024-04-06T02:49:01.918 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.932+0000 7fda3f358700 1 -- 172.21.15.105:0/123150478 shutdown_connections 2024-04-06T02:49:01.918 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.932+0000 7fda3f358700 1 --2- 172.21.15.105:0/123150478 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fda38101dc0 0x7fda38102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:01.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.932+0000 7fda3f358700 1 --2- 172.21.15.105:0/123150478 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda38100bc0 0x7fda38100fe0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:01.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.932+0000 7fda3f358700 1 -- 172.21.15.105:0/123150478 >> 172.21.15.105:0/123150478 conn(0x7fda380fc140 msgr2=0x7fda380fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:49:01.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.933+0000 7fda3f358700 1 -- 172.21.15.105:0/123150478 shutdown_connections 2024-04-06T02:49:01.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.933+0000 7fda3f358700 1 -- 172.21.15.105:0/123150478 wait complete. 2024-04-06T02:49:01.920 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.933+0000 7fda3f358700 1 Processor -- start 2024-04-06T02:49:01.921 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.933+0000 7fda3f358700 1 -- start start 2024-04-06T02:49:01.921 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.933+0000 7fda3f358700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda38100bc0 0x7fda381963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:01.921 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.934+0000 7fda3f358700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fda38101dc0 0x7fda38196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:01.921 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.934+0000 7fda3f358700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda38196f50 con 0x7fda38100bc0 2024-04-06T02:49:01.921 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.934+0000 7fda3f358700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda38197090 con 0x7fda38101dc0 2024-04-06T02:49:01.921 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.934+0000 7fda3d0f4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda38100bc0 0x7fda381963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:01.921 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.934+0000 7fda3d0f4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda38100bc0 0x7fda381963f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:51346/0 (socket says 172.21.15.105:51346) 2024-04-06T02:49:01.921 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.934+0000 7fda3d0f4700 1 -- 172.21.15.105:0/677565221 learned_addr learned my addr 172.21.15.105:0/677565221 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:49:01.921 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.934+0000 7fda3c8f3700 1 --2- 172.21.15.105:0/677565221 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fda38101dc0 0x7fda38196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:01.921 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.934+0000 7fda3d0f4700 1 -- 172.21.15.105:0/677565221 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fda38101dc0 msgr2=0x7fda38196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:01.921 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.934+0000 7fda3d0f4700 1 --2- 172.21.15.105:0/677565221 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fda38101dc0 0x7fda38196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:01.921 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.934+0000 7fda3d0f4700 1 -- 172.21.15.105:0/677565221 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda340095f0 con 0x7fda38100bc0 2024-04-06T02:49:01.921 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.934+0000 7fda3d0f4700 1 --2- 172.21.15.105:0/677565221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda38100bc0 0x7fda381963f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fda3400bda0 tx=0x7fda3400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:01.922 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.934+0000 7fda2e7fc700 1 -- 172.21.15.105:0/677565221 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda3401d070 con 0x7fda38100bc0 2024-04-06T02:49:01.922 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.934+0000 7fda3f358700 1 -- 172.21.15.105:0/677565221 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda3819bae0 con 0x7fda38100bc0 2024-04-06T02:49:01.922 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.934+0000 7fda2e7fc700 1 -- 172.21.15.105:0/677565221 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fda34022470 con 0x7fda38100bc0 2024-04-06T02:49:01.922 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.934+0000 7fda2e7fc700 1 -- 172.21.15.105:0/677565221 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda3400f650 con 0x7fda38100bc0 2024-04-06T02:49:01.922 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.934+0000 7fda3f358700 1 -- 172.21.15.105:0/677565221 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda3819c050 con 0x7fda38100bc0 2024-04-06T02:49:01.922 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.935+0000 7fda3f358700 1 -- 172.21.15.105:0/677565221 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda381097a0 con 0x7fda38100bc0 2024-04-06T02:49:01.923 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.936+0000 7fda2e7fc700 1 -- 172.21.15.105:0/677565221 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fda340225e0 con 0x7fda38100bc0 2024-04-06T02:49:01.923 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.936+0000 7fda2e7fc700 1 --2- 172.21.15.105:0/677565221 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fda2406c4e0 0x7fda2406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:01.923 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.936+0000 7fda2e7fc700 1 -- 172.21.15.105:0/677565221 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fda3408cef0 con 0x7fda38100bc0 2024-04-06T02:49:01.923 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.936+0000 7fda3c8f3700 1 --2- 172.21.15.105:0/677565221 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fda2406c4e0 0x7fda2406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:01.923 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.937+0000 7fda3c8f3700 1 --2- 172.21.15.105:0/677565221 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fda2406c4e0 0x7fda2406e9a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fda28009a10 tx=0x7fda28009190 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:01.926 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:01.941+0000 7fda2e7fc700 1 -- 172.21.15.105:0/677565221 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fda3405b1b0 con 0x7fda38100bc0 2024-04-06T02:49:02.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:02.175+0000 7fda3f358700 1 -- 172.21.15.105:0/677565221 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fda3819c690 con 0x7fda38100bc0 2024-04-06T02:49:02.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:02.176+0000 7fda2e7fc700 1 -- 172.21.15.105:0/677565221 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1838 (secure 0 0 0) 0x7fda34027740 con 0x7fda38100bc0 2024-04-06T02:49:02.162 INFO:teuthology.orchestra.run.smithi105.stdout:e14 2024-04-06T02:49:02.162 INFO:teuthology.orchestra.run.smithi105.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T02:49:02.162 INFO:teuthology.orchestra.run.smithi105.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T02:49:02.162 INFO:teuthology.orchestra.run.smithi105.stdout:legacy client fscid: 1 2024-04-06T02:49:02.162 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:49:02.162 INFO:teuthology.orchestra.run.smithi105.stdout:Filesystem 'cephfs' (1) 2024-04-06T02:49:02.163 INFO:teuthology.orchestra.run.smithi105.stdout:fs_name cephfs 2024-04-06T02:49:02.163 INFO:teuthology.orchestra.run.smithi105.stdout:epoch 14 2024-04-06T02:49:02.163 INFO:teuthology.orchestra.run.smithi105.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-06T02:49:02.163 INFO:teuthology.orchestra.run.smithi105.stdout:created 2024-04-06T02:48:41.709017+0000 2024-04-06T02:49:02.163 INFO:teuthology.orchestra.run.smithi105.stdout:modified 2024-04-06T02:49:01.525774+0000 2024-04-06T02:49:02.163 INFO:teuthology.orchestra.run.smithi105.stdout:tableserver 0 2024-04-06T02:49:02.163 INFO:teuthology.orchestra.run.smithi105.stdout:root 0 2024-04-06T02:49:02.163 INFO:teuthology.orchestra.run.smithi105.stdout:session_timeout 60 2024-04-06T02:49:02.163 INFO:teuthology.orchestra.run.smithi105.stdout:session_autoclose 300 2024-04-06T02:49:02.164 INFO:teuthology.orchestra.run.smithi105.stdout:max_file_size 1099511627776 2024-04-06T02:49:02.164 INFO:teuthology.orchestra.run.smithi105.stdout:required_client_features {} 2024-04-06T02:49:02.164 INFO:teuthology.orchestra.run.smithi105.stdout:last_failure 0 2024-04-06T02:49:02.164 INFO:teuthology.orchestra.run.smithi105.stdout:last_failure_osd_epoch 47 2024-04-06T02:49:02.164 INFO:teuthology.orchestra.run.smithi105.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-06T02:49:02.164 INFO:teuthology.orchestra.run.smithi105.stdout:max_mds 2 2024-04-06T02:49:02.165 INFO:teuthology.orchestra.run.smithi105.stdout:in 0,1 2024-04-06T02:49:02.165 INFO:teuthology.orchestra.run.smithi105.stdout:up {0=14486,1=24259} 2024-04-06T02:49:02.165 INFO:teuthology.orchestra.run.smithi105.stdout:failed 2024-04-06T02:49:02.165 INFO:teuthology.orchestra.run.smithi105.stdout:damaged 2024-04-06T02:49:02.165 INFO:teuthology.orchestra.run.smithi105.stdout:stopped 2024-04-06T02:49:02.165 INFO:teuthology.orchestra.run.smithi105.stdout:data_pools [3] 2024-04-06T02:49:02.165 INFO:teuthology.orchestra.run.smithi105.stdout:metadata_pool 2 2024-04-06T02:49:02.166 INFO:teuthology.orchestra.run.smithi105.stdout:inline_data enabled 2024-04-06T02:49:02.166 INFO:teuthology.orchestra.run.smithi105.stdout:balancer 2024-04-06T02:49:02.166 INFO:teuthology.orchestra.run.smithi105.stdout:bal_rank_mask -1 2024-04-06T02:49:02.166 INFO:teuthology.orchestra.run.smithi105.stdout:standby_count_wanted 1 2024-04-06T02:49:02.166 INFO:teuthology.orchestra.run.smithi105.stdout:[mds.cephfs.smithi105.oxxcbq{0:14486} state up:rejoin seq 5 join_fscid=1 addr [v2:172.21.15.105:6828/2512987070,v1:172.21.15.105:6829/2512987070] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T02:49:02.166 INFO:teuthology.orchestra.run.smithi105.stdout:[mds.cephfs.smithi184.oochxp{1:24259} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.184:6824/2770403500,v1:172.21.15.184:6825/2770403500] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T02:49:02.166 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:49:02.166 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:49:02.167 INFO:teuthology.orchestra.run.smithi105.stdout:Standby daemons: 2024-04-06T02:49:02.167 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:49:02.167 INFO:teuthology.orchestra.run.smithi105.stdout:[mds.cephfs.smithi105.mrivth{-1:14496} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.105:6826/2109838618,v1:172.21.15.105:6827/2109838618] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T02:49:02.167 INFO:teuthology.orchestra.run.smithi105.stdout:[mds.cephfs.smithi184.htgrjn{-1:24265} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.184:6826/323195824,v1:172.21.15.184:6827/323195824] compat {c=[1],r=[1],i=[7ff]}] 2024-04-06T02:49:02.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:02.178+0000 7fda3f358700 1 -- 172.21.15.105:0/677565221 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fda2406c4e0 msgr2=0x7fda2406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:02.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:02.178+0000 7fda3f358700 1 --2- 172.21.15.105:0/677565221 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fda2406c4e0 0x7fda2406e9a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fda28009a10 tx=0x7fda28009190 comp rx=0 tx=0).stop 2024-04-06T02:49:02.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:02.178+0000 7fda3f358700 1 -- 172.21.15.105:0/677565221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda38100bc0 msgr2=0x7fda381963f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:02.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:02.178+0000 7fda3f358700 1 --2- 172.21.15.105:0/677565221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda38100bc0 0x7fda381963f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fda3400bda0 tx=0x7fda3400bdd0 comp rx=0 tx=0).stop 2024-04-06T02:49:02.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:02.179+0000 7fda3f358700 1 -- 172.21.15.105:0/677565221 shutdown_connections 2024-04-06T02:49:02.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:02.179+0000 7fda3f358700 1 --2- 172.21.15.105:0/677565221 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fda2406c4e0 0x7fda2406e9a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:02.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:02.179+0000 7fda3f358700 1 --2- 172.21.15.105:0/677565221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda38100bc0 0x7fda381963f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:02.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:02.179+0000 7fda3f358700 1 --2- 172.21.15.105:0/677565221 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fda38101dc0 0x7fda38196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:02.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:02.179+0000 7fda3f358700 1 -- 172.21.15.105:0/677565221 >> 172.21.15.105:0/677565221 conn(0x7fda380fc140 msgr2=0x7fda38104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:49:02.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:02.179+0000 7fda3f358700 1 -- 172.21.15.105:0/677565221 shutdown_connections 2024-04-06T02:49:02.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:02.179+0000 7fda3f358700 1 -- 172.21.15.105:0/677565221 wait complete. 2024-04-06T02:49:02.169 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 14 2024-04-06T02:49:02.866 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:02 smithi105 ceph-mon[29228]: mds.? [v2:172.21.15.105:6828/2512987070,v1:172.21.15.105:6829/2512987070] up:rejoin 2024-04-06T02:49:02.867 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:02 smithi105 ceph-mon[29228]: fsmap cephfs:2/2 {0=cephfs.smithi105.oxxcbq=up:rejoin,1=cephfs.smithi184.oochxp=up:active} 2 up:standby 2024-04-06T02:49:02.867 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:02 smithi105 ceph-mon[29228]: daemon mds.cephfs.smithi105.oxxcbq is now active in filesystem cephfs as rank 0 2024-04-06T02:49:02.867 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:02 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/677565221' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T02:49:02.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:02 smithi184 ceph-mon[37439]: mds.? [v2:172.21.15.105:6828/2512987070,v1:172.21.15.105:6829/2512987070] up:rejoin 2024-04-06T02:49:02.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:02 smithi184 ceph-mon[37439]: fsmap cephfs:2/2 {0=cephfs.smithi105.oxxcbq=up:rejoin,1=cephfs.smithi184.oochxp=up:active} 2 up:standby 2024-04-06T02:49:02.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:02 smithi184 ceph-mon[37439]: daemon mds.cephfs.smithi105.oxxcbq is now active in filesystem cephfs as rank 0 2024-04-06T02:49:02.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:02 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/677565221' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-06T02:49:03.215 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-04-06T02:49:03.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:03 smithi105 ceph-mon[29228]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-06T02:49:03.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:03 smithi105 ceph-mon[29228]: mds.? [v2:172.21.15.105:6828/2512987070,v1:172.21.15.105:6829/2512987070] up:active 2024-04-06T02:49:03.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:03 smithi105 ceph-mon[29228]: fsmap cephfs:2 {0=cephfs.smithi105.oxxcbq=up:active,1=cephfs.smithi184.oochxp=up:active} 2 up:standby 2024-04-06T02:49:03.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:03 smithi105 ceph-mon[29228]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 102 B/s wr, 2 op/s 2024-04-06T02:49:03.963 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:49:03.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:03 smithi184 ceph-mon[37439]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-06T02:49:03.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:03 smithi184 ceph-mon[37439]: mds.? [v2:172.21.15.105:6828/2512987070,v1:172.21.15.105:6829/2512987070] up:active 2024-04-06T02:49:03.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:03 smithi184 ceph-mon[37439]: fsmap cephfs:2 {0=cephfs.smithi105.oxxcbq=up:active,1=cephfs.smithi184.oochxp=up:active} 2 up:standby 2024-04-06T02:49:03.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:03 smithi184 ceph-mon[37439]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 102 B/s wr, 2 op/s 2024-04-06T02:49:05.714 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.725+0000 7f3396669700 1 -- 172.21.15.105:0/334031514 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33900ffab0 msgr2=0x7f33900ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:05.714 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.725+0000 7f3396669700 1 --2- 172.21.15.105:0/334031514 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33900ffab0 0x7f33900ffed0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3380009910 tx=0x7f3380009c20 comp rx=0 tx=0).stop 2024-04-06T02:49:05.714 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.725+0000 7f3396669700 1 -- 172.21.15.105:0/334031514 shutdown_connections 2024-04-06T02:49:05.714 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.725+0000 7f3396669700 1 --2- 172.21.15.105:0/334031514 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3390100cb0 0x7f3390101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:05.714 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.725+0000 7f3396669700 1 --2- 172.21.15.105:0/334031514 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33900ffab0 0x7f33900ffed0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:05.714 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.725+0000 7f3396669700 1 -- 172.21.15.105:0/334031514 >> 172.21.15.105:0/334031514 conn(0x7f33900fb050 msgr2=0x7f33900fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:49:05.714 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.725+0000 7f3396669700 1 -- 172.21.15.105:0/334031514 shutdown_connections 2024-04-06T02:49:05.714 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.725+0000 7f3396669700 1 -- 172.21.15.105:0/334031514 wait complete. 2024-04-06T02:49:05.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.726+0000 7f3396669700 1 Processor -- start 2024-04-06T02:49:05.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.726+0000 7f3396669700 1 -- start start 2024-04-06T02:49:05.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.726+0000 7f3396669700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33900ffab0 0x7f3390194150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:05.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.726+0000 7f3396669700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3390100cb0 0x7f3390194690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:05.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.726+0000 7f3396669700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3390194cb0 con 0x7f33900ffab0 2024-04-06T02:49:05.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.726+0000 7f3396669700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3390194df0 con 0x7f3390100cb0 2024-04-06T02:49:05.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.726+0000 7f338ffff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33900ffab0 0x7f3390194150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:05.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.727+0000 7f338f7fe700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3390100cb0 0x7f3390194690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:05.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.727+0000 7f338f7fe700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3390100cb0 0x7f3390194690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.105:48378/0 (socket says 172.21.15.105:48378) 2024-04-06T02:49:05.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.727+0000 7f338f7fe700 1 -- 172.21.15.105:0/484542795 learned_addr learned my addr 172.21.15.105:0/484542795 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:49:05.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.727+0000 7f338ffff700 1 -- 172.21.15.105:0/484542795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3390100cb0 msgr2=0x7f3390194690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:05.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.727+0000 7f338ffff700 1 --2- 172.21.15.105:0/484542795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3390100cb0 0x7f3390194690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:05.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.727+0000 7f338ffff700 1 -- 172.21.15.105:0/484542795 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33800095f0 con 0x7f33900ffab0 2024-04-06T02:49:05.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.727+0000 7f338ffff700 1 --2- 172.21.15.105:0/484542795 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33900ffab0 0x7f3390194150 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3380000c00 tx=0x7f338000fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:05.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.727+0000 7f338d7fa700 1 -- 172.21.15.105:0/484542795 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f338001d070 con 0x7f33900ffab0 2024-04-06T02:49:05.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.727+0000 7f3396669700 1 -- 172.21.15.105:0/484542795 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3390199840 con 0x7f33900ffab0 2024-04-06T02:49:05.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.727+0000 7f338d7fa700 1 -- 172.21.15.105:0/484542795 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3380005dc0 con 0x7f33900ffab0 2024-04-06T02:49:05.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.727+0000 7f3396669700 1 -- 172.21.15.105:0/484542795 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3390199d00 con 0x7f33900ffab0 2024-04-06T02:49:05.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.727+0000 7f338d7fa700 1 -- 172.21.15.105:0/484542795 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3380022470 con 0x7f33900ffab0 2024-04-06T02:49:05.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.728+0000 7f3396669700 1 -- 172.21.15.105:0/484542795 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f337c005320 con 0x7f33900ffab0 2024-04-06T02:49:05.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.729+0000 7f338d7fa700 1 -- 172.21.15.105:0/484542795 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f3380005160 con 0x7f33900ffab0 2024-04-06T02:49:05.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.729+0000 7f338d7fa700 1 --2- 172.21.15.105:0/484542795 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f337806e6f0 0x7f3378070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:05.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.729+0000 7f338f7fe700 1 --2- 172.21.15.105:0/484542795 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f337806e6f0 0x7f3378070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:05.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.729+0000 7f338d7fa700 1 -- 172.21.15.105:0/484542795 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f338008cce0 con 0x7f33900ffab0 2024-04-06T02:49:05.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.730+0000 7f338f7fe700 1 --2- 172.21.15.105:0/484542795 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f337806e6f0 0x7f3378070bb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3384009d20 tx=0x7f3384009260 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:05.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:05.734+0000 7f338d7fa700 1 -- 172.21.15.105:0/484542795 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f338005aef0 con 0x7f33900ffab0 2024-04-06T02:49:05.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:05 smithi105 ceph-mon[29228]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 102 B/s wr, 2 op/s 2024-04-06T02:49:05.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:05 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:49:05.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:05 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:49:05.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:05 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:49:05.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:05 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:49:05.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:05 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:49:05.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:05 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:49:05.945 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:05 smithi184 ceph-mon[37439]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 102 B/s wr, 2 op/s 2024-04-06T02:49:05.945 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:05 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:49:05.945 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:05 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:49:05.945 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:05 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:49:05.945 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:05 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:49:05.945 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:05 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:49:05.946 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:05 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:49:06.003 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:06.018+0000 7f3396669700 1 -- 172.21.15.105:0/484542795 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f337c005f70 con 0x7f33900ffab0 2024-04-06T02:49:06.004 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:06.019+0000 7f338d7fa700 1 -- 172.21.15.105:0/484542795 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4763 (secure 0 0 0) 0x7f338002a350 con 0x7f33900ffab0 2024-04-06T02:49:06.006 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:06.021+0000 7f3396669700 1 -- 172.21.15.105:0/484542795 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f337806e6f0 msgr2=0x7f3378070bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:06.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:06.021+0000 7f3396669700 1 --2- 172.21.15.105:0/484542795 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f337806e6f0 0x7f3378070bb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3384009d20 tx=0x7f3384009260 comp rx=0 tx=0).stop 2024-04-06T02:49:06.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:06.021+0000 7f3396669700 1 -- 172.21.15.105:0/484542795 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33900ffab0 msgr2=0x7f3390194150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:06.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:06.021+0000 7f3396669700 1 --2- 172.21.15.105:0/484542795 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33900ffab0 0x7f3390194150 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3380000c00 tx=0x7f338000fa00 comp rx=0 tx=0).stop 2024-04-06T02:49:06.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:06.021+0000 7f3396669700 1 -- 172.21.15.105:0/484542795 shutdown_connections 2024-04-06T02:49:06.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:06.021+0000 7f3396669700 1 --2- 172.21.15.105:0/484542795 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f337806e6f0 0x7f3378070bb0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:06.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:06.021+0000 7f3396669700 1 --2- 172.21.15.105:0/484542795 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33900ffab0 0x7f3390194150 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:06.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:06.021+0000 7f3396669700 1 --2- 172.21.15.105:0/484542795 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3390100cb0 0x7f3390194690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:06.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:06.021+0000 7f3396669700 1 -- 172.21.15.105:0/484542795 >> 172.21.15.105:0/484542795 conn(0x7f33900fb050 msgr2=0x7f3390103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:49:06.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:06.021+0000 7f3396669700 1 -- 172.21.15.105:0/484542795 shutdown_connections 2024-04-06T02:49:06.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:06.021+0000 7f3396669700 1 -- 172.21.15.105:0/484542795 wait complete. 2024-04-06T02:49:06.008 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 15 2024-04-06T02:49:06.017 INFO:teuthology.orchestra.run.smithi105.stdout:true 2024-04-06T02:49:06.736 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:06 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/484542795' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-06T02:49:06.759 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-04-06T02:49:06.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:06 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/484542795' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-06T02:49:07.303 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:49:07.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:07 smithi184 ceph-mon[37439]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 4.6 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-06T02:49:07.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:07 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:49:07.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:07 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:49:08.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:07 smithi105 ceph-mon[29228]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 4.6 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-06T02:49:08.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:07 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:49:08.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:07 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:49:09.689 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.703+0000 7fae535fd700 1 -- 172.21.15.105:0/2037229727 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae4c101dd0 msgr2=0x7fae4c102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:09.689 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.703+0000 7fae535fd700 1 --2- 172.21.15.105:0/2037229727 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae4c101dd0 0x7fae4c102230 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fae3c009910 tx=0x7fae3c009c20 comp rx=0 tx=0).stop 2024-04-06T02:49:09.690 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.703+0000 7fae535fd700 1 -- 172.21.15.105:0/2037229727 shutdown_connections 2024-04-06T02:49:09.690 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.703+0000 7fae535fd700 1 --2- 172.21.15.105:0/2037229727 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae4c101dd0 0x7fae4c102230 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:09.690 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.703+0000 7fae535fd700 1 --2- 172.21.15.105:0/2037229727 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae4c100bd0 0x7fae4c100ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:09.690 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.703+0000 7fae535fd700 1 -- 172.21.15.105:0/2037229727 >> 172.21.15.105:0/2037229727 conn(0x7fae4c0fc170 msgr2=0x7fae4c0fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:49:09.690 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.703+0000 7fae535fd700 1 -- 172.21.15.105:0/2037229727 shutdown_connections 2024-04-06T02:49:09.690 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.703+0000 7fae535fd700 1 -- 172.21.15.105:0/2037229727 wait complete. 2024-04-06T02:49:09.690 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.704+0000 7fae535fd700 1 Processor -- start 2024-04-06T02:49:09.690 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.704+0000 7fae535fd700 1 -- start start 2024-04-06T02:49:09.690 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.704+0000 7fae535fd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae4c100bd0 0x7fae4c074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:09.690 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.704+0000 7fae535fd700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae4c101dd0 0x7fae4c073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:09.690 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.704+0000 7fae535fd700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c0736c0 con 0x7fae4c100bd0 2024-04-06T02:49:09.690 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.704+0000 7fae535fd700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c073800 con 0x7fae4c101dd0 2024-04-06T02:49:09.691 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.704+0000 7fae51399700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae4c100bd0 0x7fae4c074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:09.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.704+0000 7fae51399700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae4c100bd0 0x7fae4c074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:51370/0 (socket says 172.21.15.105:51370) 2024-04-06T02:49:09.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.704+0000 7fae51399700 1 -- 172.21.15.105:0/1453116445 learned_addr learned my addr 172.21.15.105:0/1453116445 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:49:09.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.704+0000 7fae51399700 1 -- 172.21.15.105:0/1453116445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae4c101dd0 msgr2=0x7fae4c073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:09.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.704+0000 7fae51399700 1 --2- 172.21.15.105:0/1453116445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae4c101dd0 0x7fae4c073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:09.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.704+0000 7fae51399700 1 -- 172.21.15.105:0/1453116445 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3c0095f0 con 0x7fae4c100bd0 2024-04-06T02:49:09.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.705+0000 7fae51399700 1 --2- 172.21.15.105:0/1453116445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae4c100bd0 0x7fae4c074b30 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fae4800d900 tx=0x7fae4800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:09.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.705+0000 7fae427fc700 1 -- 172.21.15.105:0/1453116445 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae480041f0 con 0x7fae4c100bd0 2024-04-06T02:49:09.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.705+0000 7fae427fc700 1 -- 172.21.15.105:0/1453116445 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fae48004350 con 0x7fae4c100bd0 2024-04-06T02:49:09.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.705+0000 7fae535fd700 1 -- 172.21.15.105:0/1453116445 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae4c073ae0 con 0x7fae4c100bd0 2024-04-06T02:49:09.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.705+0000 7fae427fc700 1 -- 172.21.15.105:0/1453116445 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae48003da0 con 0x7fae4c100bd0 2024-04-06T02:49:09.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.705+0000 7fae535fd700 1 -- 172.21.15.105:0/1453116445 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae4c19e3e0 con 0x7fae4c100bd0 2024-04-06T02:49:09.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.706+0000 7fae535fd700 1 -- 172.21.15.105:0/1453116445 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae4c109730 con 0x7fae4c100bd0 2024-04-06T02:49:09.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.707+0000 7fae427fc700 1 -- 172.21.15.105:0/1453116445 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fae48009940 con 0x7fae4c100bd0 2024-04-06T02:49:09.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.707+0000 7fae427fc700 1 --2- 172.21.15.105:0/1453116445 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fae3806c4e0 0x7fae3806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:09.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.708+0000 7fae427fc700 1 -- 172.21.15.105:0/1453116445 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fae4808b150 con 0x7fae4c100bd0 2024-04-06T02:49:09.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.708+0000 7fae50b98700 1 --2- 172.21.15.105:0/1453116445 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fae3806c4e0 0x7fae3806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:09.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.708+0000 7fae50b98700 1 --2- 172.21.15.105:0/1453116445 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fae3806c4e0 0x7fae3806e9a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fae3c00b5c0 tx=0x7fae3c005960 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:09.698 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:09.712+0000 7fae427fc700 1 -- 172.21.15.105:0/1453116445 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fae48059480 con 0x7fae4c100bd0 2024-04-06T02:49:09.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:09 smithi105 ceph-mon[29228]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.1 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-06T02:49:09.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:09 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:49:09.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:09 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:49:09.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:09 smithi184 ceph-mon[37439]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.1 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-06T02:49:09.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:09 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:49:09.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:09 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:49:10.001 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.016+0000 7fae535fd700 1 -- 172.21.15.105:0/1453116445 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7fae4c1098f0 con 0x7fae4c100bd0 2024-04-06T02:49:10.001 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.016+0000 7fae427fc700 1 -- 172.21.15.105:0/1453116445 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v15) v1 ==== 78+0+83 (secure 0 0 0) 0x7fae48059010 con 0x7fae4c100bd0 2024-04-06T02:49:10.004 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.018+0000 7fae535fd700 1 -- 172.21.15.105:0/1453116445 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fae3806c4e0 msgr2=0x7fae3806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:10.004 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.018+0000 7fae535fd700 1 --2- 172.21.15.105:0/1453116445 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fae3806c4e0 0x7fae3806e9a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fae3c00b5c0 tx=0x7fae3c005960 comp rx=0 tx=0).stop 2024-04-06T02:49:10.004 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.018+0000 7fae535fd700 1 -- 172.21.15.105:0/1453116445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae4c100bd0 msgr2=0x7fae4c074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:10.004 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.018+0000 7fae535fd700 1 --2- 172.21.15.105:0/1453116445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae4c100bd0 0x7fae4c074b30 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fae4800d900 tx=0x7fae4800dcc0 comp rx=0 tx=0).stop 2024-04-06T02:49:10.004 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.018+0000 7fae535fd700 1 -- 172.21.15.105:0/1453116445 shutdown_connections 2024-04-06T02:49:10.004 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.018+0000 7fae535fd700 1 --2- 172.21.15.105:0/1453116445 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fae3806c4e0 0x7fae3806e9a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:10.004 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.018+0000 7fae535fd700 1 --2- 172.21.15.105:0/1453116445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae4c100bd0 0x7fae4c074b30 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:10.004 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.018+0000 7fae535fd700 1 --2- 172.21.15.105:0/1453116445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae4c101dd0 0x7fae4c073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:10.004 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.018+0000 7fae535fd700 1 -- 172.21.15.105:0/1453116445 >> 172.21.15.105:0/1453116445 conn(0x7fae4c0fc170 msgr2=0x7fae4c105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:49:10.005 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.018+0000 7fae535fd700 1 -- 172.21.15.105:0/1453116445 shutdown_connections 2024-04-06T02:49:10.005 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.018+0000 7fae535fd700 1 -- 172.21.15.105:0/1453116445 wait complete. 2024-04-06T02:49:10.014 INFO:teuthology.orchestra.run.smithi105.stdout:true 2024-04-06T02:49:10.698 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-04-06T02:49:10.712 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T02:49:10.837 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.850+0000 7f6ca4166640 1 -- 172.21.15.105:0/125929675 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c9c1012c0 msgr2=0x7f6c9c101740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:10.838 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.850+0000 7f6ca4166640 1 --2- 172.21.15.105:0/125929675 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c9c1012c0 0x7f6c9c101740 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c009880 tx=0x7f6c8c02bce0 comp rx=0 tx=0).stop 2024-04-06T02:49:10.838 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.850+0000 7f6ca4166640 1 -- 172.21.15.105:0/125929675 shutdown_connections 2024-04-06T02:49:10.838 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.850+0000 7f6ca4166640 1 --2- 172.21.15.105:0/125929675 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c9c1012c0 0x7f6c9c101740 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:10.838 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.850+0000 7f6ca4166640 1 --2- 172.21.15.105:0/125929675 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c9c1000d0 0x7f6c9c1004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:10.838 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.850+0000 7f6ca4166640 1 -- 172.21.15.105:0/125929675 >> 172.21.15.105:0/125929675 conn(0x7f6c9c0fb640 msgr2=0x7f6c9c0fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:49:10.838 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.850+0000 7f6ca4166640 1 -- 172.21.15.105:0/125929675 shutdown_connections 2024-04-06T02:49:10.838 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.850+0000 7f6ca4166640 1 -- 172.21.15.105:0/125929675 wait complete. 2024-04-06T02:49:10.838 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.851+0000 7f6ca4166640 1 Processor -- start 2024-04-06T02:49:10.838 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.851+0000 7f6ca4166640 1 -- start start 2024-04-06T02:49:10.838 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.851+0000 7f6ca4166640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c9c1000d0 0x7f6c9c1958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:10.838 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.851+0000 7f6ca4166640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c9c1012c0 0x7f6c9c195e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:10.838 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.851+0000 7f6ca4166640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c1963f0 con 0x7f6c9c1000d0 2024-04-06T02:49:10.839 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.851+0000 7f6ca4166640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c196560 con 0x7f6c9c1012c0 2024-04-06T02:49:10.839 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.851+0000 7f6ca1f02640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c9c1000d0 0x7f6c9c1958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:10.839 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.851+0000 7f6ca1f02640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c9c1000d0 0x7f6c9c1958e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50986/0 (socket says 172.21.15.105:50986) 2024-04-06T02:49:10.839 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.851+0000 7f6ca1f02640 1 -- 172.21.15.105:0/2402301678 learned_addr learned my addr 172.21.15.105:0/2402301678 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:49:10.839 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.851+0000 7f6ca1701640 1 --2- 172.21.15.105:0/2402301678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c9c1012c0 0x7f6c9c195e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:10.839 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.851+0000 7f6ca1f02640 1 -- 172.21.15.105:0/2402301678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c9c1012c0 msgr2=0x7f6c9c195e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:10.839 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.851+0000 7f6ca1f02640 1 --2- 172.21.15.105:0/2402301678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c9c1012c0 0x7f6c9c195e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:10.839 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.851+0000 7f6ca1f02640 1 -- 172.21.15.105:0/2402301678 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c8c009560 con 0x7f6c9c1000d0 2024-04-06T02:49:10.839 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.852+0000 7f6ca1f02640 1 --2- 172.21.15.105:0/2402301678 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c9c1000d0 0x7f6c9c1958e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6c9800b3d0 tx=0x7f6c9800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:10.839 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.852+0000 7f6c92ffd640 1 -- 172.21.15.105:0/2402301678 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c98011800 con 0x7f6c9c1000d0 2024-04-06T02:49:10.839 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.852+0000 7f6c92ffd640 1 -- 172.21.15.105:0/2402301678 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6c98011e10 con 0x7f6c9c1000d0 2024-04-06T02:49:10.839 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.852+0000 7f6c92ffd640 1 -- 172.21.15.105:0/2402301678 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c9800f510 con 0x7f6c9c1000d0 2024-04-06T02:49:10.839 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.852+0000 7f6ca4166640 1 -- 172.21.15.105:0/2402301678 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c9c19aff0 con 0x7f6c9c1000d0 2024-04-06T02:49:10.839 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.852+0000 7f6ca4166640 1 -- 172.21.15.105:0/2402301678 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c9c19b570 con 0x7f6c9c1000d0 2024-04-06T02:49:10.840 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.853+0000 7f6ca4166640 1 -- 172.21.15.105:0/2402301678 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c9c108ee0 con 0x7f6c9c1000d0 2024-04-06T02:49:10.840 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.853+0000 7f6c92ffd640 1 -- 172.21.15.105:0/2402301678 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f6c98010380 con 0x7f6c9c1000d0 2024-04-06T02:49:10.840 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.854+0000 7f6c92ffd640 1 --2- 172.21.15.105:0/2402301678 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f6c8806c510 0x7f6c8806e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:10.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.854+0000 7f6c92ffd640 1 -- 172.21.15.105:0/2402301678 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f6c9808b5b0 con 0x7f6c9c1000d0 2024-04-06T02:49:10.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.854+0000 7f6ca1701640 1 --2- 172.21.15.105:0/2402301678 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f6c8806c510 0x7f6c8806e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:10.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.854+0000 7f6ca1701640 1 --2- 172.21.15.105:0/2402301678 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f6c8806c510 0x7f6c8806e9d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c196df0 tx=0x7f6c8c000bc0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:10.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:10.858+0000 7f6c92ffd640 1 -- 172.21.15.105:0/2402301678 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6c98055db0 con 0x7f6c9c1000d0 2024-04-06T02:49:10.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:10 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/1453116445' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-06T02:49:10.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:10 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/1453116445' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-06T02:49:11.015 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.030+0000 7f6ca4166640 1 -- 172.21.15.105:0/2402301678 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c9c19b840 con 0x7f6c9c1000d0 2024-04-06T02:49:11.017 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.031+0000 7f6c92ffd640 1 -- 172.21.15.105:0/2402301678 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+15194 (secure 0 0 0) 0x7f6c980593d0 con 0x7f6c9c1000d0 2024-04-06T02:49:11.017 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:49:11.017 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":47,"fsid":"130df578-f3bf-11ee-b649-cb9ed24678a4","created":"2024-04-06T02:41:32.644992+0000","modified":"2024-04-06T02:48:58.511589+0000","last_up_change":"2024-04-06T02:47:25.576978+0000","last_in_change":"2024-04-06T02:47:06.634446+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T02:46:12.900794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-06T02:48:40.578190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-06T02:48:40.940043+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41c20c92-5314-4354-b659-cb449ef98c81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2018109658},{"type":"v1","addr":"172.21.15.105:6803","nonce":2018109658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2018109658},{"type":"v1","addr":"172.21.15.105:6805","nonce":2018109658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":2018109658},{"type":"v1","addr":"172.21.15.105:6809","nonce":2018109658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2018109658},{"type":"v1","addr":"172.21.15.105:6807","nonce":2018109658}]},"public_addr":"172.21.15.105:6803/2018109658","cluster_addr":"172.21.15.105:6805/2018109658","heartbeat_back_addr":"172.21.15.105:6809/2018109658","heartbeat_front_addr":"172.21.15.105:6807/2018109658","state":["exists","up"]},{"osd":1,"uuid":"ea3f1b19-62f8-488e-b62a-2340aef1112a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":1345191629},{"type":"v1","addr":"172.21.15.105:6811","nonce":1345191629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":1345191629},{"type":"v1","addr":"172.21.15.105:6813","nonce":1345191629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":1345191629},{"type":"v1","addr":"172.21.15.105:6817","nonce":1345191629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":1345191629},{"type":"v1","addr":"172.21.15.105:6815","nonce":1345191629}]},"public_addr":"172.21.15.105:6811/1345191629","cluster_addr":"172.21.15.105:6813/1345191629","heartbeat_back_addr":"172.21.15.105:6817/1345191629","heartbeat_front_addr":"172.21.15.105:6815/1345191629","state":["exists","up"]},{"osd":2,"uuid":"def7693a-bb30-4c74-ae20-59a7a86ef9db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":3651690266},{"type":"v1","addr":"172.21.15.105:6819","nonce":3651690266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":3651690266},{"type":"v1","addr":"172.21.15.105:6821","nonce":3651690266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3651690266},{"type":"v1","addr":"172.21.15.105:6825","nonce":3651690266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":3651690266},{"type":"v1","addr":"172.21.15.105:6823","nonce":3651690266}]},"public_addr":"172.21.15.105:6819/3651690266","cluster_addr":"172.21.15.105:6821/3651690266","heartbeat_back_addr":"172.21.15.105:6825/3651690266","heartbeat_front_addr":"172.21.15.105:6823/3651690266","state":["exists","up"]},{"osd":3,"uuid":"97f959d7-5922-4423-9d63-e2fe677a7019","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":330017535},{"type":"v1","addr":"172.21.15.184:6801","nonce":330017535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":330017535},{"type":"v1","addr":"172.21.15.184:6803","nonce":330017535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":330017535},{"type":"v1","addr":"172.21.15.184:6807","nonce":330017535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":330017535},{"type":"v1","addr":"172.21.15.184:6805","nonce":330017535}]},"public_addr":"172.21.15.184:6801/330017535","cluster_addr":"172.21.15.184:6803/330017535","heartbeat_back_addr":"172.21.15.184:6807/330017535","heartbeat_front_addr":"172.21.15.184:6805/330017535","state":["exists","up"]},{"osd":4,"uuid":"34d7ebbd-c725-4948-ae33-3ffb40ec9979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2237051966},{"type":"v1","addr":"172.21.15.184:6809","nonce":2237051966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2237051966},{"type":"v1","addr":"172.21.15.184:6811","nonce":2237051966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2237051966},{"type":"v1","addr":"172.21.15.184:6815","nonce":2237051966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2237051966},{"type":"v1","addr":"172.21.15.184:6813","nonce":2237051966}]},"public_addr":"172.21.15.184:6809/2237051966","cluster_addr":"172.21.15.184:6811/2237051966","heartbeat_back_addr":"172.21.15.184:6815/2237051966","heartbeat_front_addr":"172.21.15.184:6813/2237051966","state":["exists","up"]},{"osd":5,"uuid":"95e64c1c-1f14-4017-b27c-11f447ac5a67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":1371998009},{"type":"v1","addr":"172.21.15.184:6817","nonce":1371998009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":1371998009},{"type":"v1","addr":"172.21.15.184:6819","nonce":1371998009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":1371998009},{"type":"v1","addr":"172.21.15.184:6823","nonce":1371998009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":1371998009},{"type":"v1","addr":"172.21.15.184:6821","nonce":1371998009}]},"public_addr":"172.21.15.184:6817/1371998009","cluster_addr":"172.21.15.184:6819/1371998009","heartbeat_back_addr":"172.21.15.184:6823/1371998009","heartbeat_front_addr":"172.21.15.184:6821/1371998009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:45:12.515625+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:45:41.117291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:46:09.180416+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:46:33.131470+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:46:58.464606+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T02:47:24.027273+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:6827/2482683361":"2024-04-07T02:48:58.511566+0000","172.21.15.105:0/3833744400":"2024-04-07T02:42:35.032003+0000","172.21.15.105:6801/1949998385":"2024-04-07T02:43:54.575029+0000","172.21.15.105:6826/2482683361":"2024-04-07T02:48:58.511566+0000","172.21.15.105:6800/3245193613":"2024-04-07T02:42:02.154709+0000","172.21.15.105:0/403675292":"2024-04-07T02:43:54.575029+0000","172.21.15.105:6801/3245193613":"2024-04-07T02:42:02.154709+0000","172.21.15.105:0/1826925654":"2024-04-07T02:42:02.154709+0000","172.21.15.105:0/1323549385":"2024-04-07T02:42:02.154709+0000","172.21.15.105:0/1539671918":"2024-04-07T02:42:02.154709+0000","172.21.15.105:0/2131480441":"2024-04-07T02:42:35.032003+0000","172.21.15.105:0/1155423922":"2024-04-07T02:42:35.032003+0000","172.21.15.105:6800/1394104199":"2024-04-07T02:42:35.032003+0000","172.21.15.105:6801/1394104199":"2024-04-07T02:42:35.032003+0000","172.21.15.105:6800/1949998385":"2024-04-07T02:43:54.575029+0000","172.21.15.105:0/1342777414":"2024-04-07T02:43:54.575029+0000","172.21.15.105:0/2231570326":"2024-04-07T02:43:54.575029+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T02:49:11.019 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.033+0000 7f6ca4166640 1 -- 172.21.15.105:0/2402301678 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f6c8806c510 msgr2=0x7f6c8806e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:11.019 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.033+0000 7f6ca4166640 1 --2- 172.21.15.105:0/2402301678 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f6c8806c510 0x7f6c8806e9d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c196df0 tx=0x7f6c8c000bc0 comp rx=0 tx=0).stop 2024-04-06T02:49:11.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.033+0000 7f6ca4166640 1 -- 172.21.15.105:0/2402301678 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c9c1000d0 msgr2=0x7f6c9c1958e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:11.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.033+0000 7f6ca4166640 1 --2- 172.21.15.105:0/2402301678 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c9c1000d0 0x7f6c9c1958e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6c9800b3d0 tx=0x7f6c9800b890 comp rx=0 tx=0).stop 2024-04-06T02:49:11.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.034+0000 7f6ca4166640 1 -- 172.21.15.105:0/2402301678 shutdown_connections 2024-04-06T02:49:11.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.034+0000 7f6ca4166640 1 --2- 172.21.15.105:0/2402301678 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f6c8806c510 0x7f6c8806e9d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:11.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.034+0000 7f6ca4166640 1 --2- 172.21.15.105:0/2402301678 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c9c1012c0 0x7f6c9c195e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:11.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.034+0000 7f6ca4166640 1 --2- 172.21.15.105:0/2402301678 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c9c1000d0 0x7f6c9c1958e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:11.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.034+0000 7f6ca4166640 1 -- 172.21.15.105:0/2402301678 >> 172.21.15.105:0/2402301678 conn(0x7f6c9c0fb640 msgr2=0x7f6c9c1044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:49:11.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.034+0000 7f6ca4166640 1 -- 172.21.15.105:0/2402301678 shutdown_connections 2024-04-06T02:49:11.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.034+0000 7f6ca4166640 1 -- 172.21.15.105:0/2402301678 wait complete. 2024-04-06T02:49:11.032 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T02:46:12.900794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-06T02:48:40.578190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-06T02:48:40.940043+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T02:49:11.032 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-06T02:49:11.147 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.160+0000 7f3e92549640 1 -- 172.21.15.105:0/1096282523 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e8c100b80 msgr2=0x7f3e8c102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:11.147 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.160+0000 7f3e92549640 1 --2- 172.21.15.105:0/1096282523 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e8c100b80 0x7f3e8c102f70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3e80009880 tx=0x7f3e8002bce0 comp rx=0 tx=0).stop 2024-04-06T02:49:11.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.160+0000 7f3e92549640 1 -- 172.21.15.105:0/1096282523 shutdown_connections 2024-04-06T02:49:11.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.160+0000 7f3e92549640 1 --2- 172.21.15.105:0/1096282523 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e8c1034b0 0x7f3e8c1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:11.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.160+0000 7f3e92549640 1 --2- 172.21.15.105:0/1096282523 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e8c100b80 0x7f3e8c102f70 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:11.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.160+0000 7f3e92549640 1 -- 172.21.15.105:0/1096282523 >> 172.21.15.105:0/1096282523 conn(0x7f3e8c0fa530 msgr2=0x7f3e8c0fc950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:49:11.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.160+0000 7f3e92549640 1 -- 172.21.15.105:0/1096282523 shutdown_connections 2024-04-06T02:49:11.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.160+0000 7f3e92549640 1 -- 172.21.15.105:0/1096282523 wait complete. 2024-04-06T02:49:11.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.160+0000 7f3e92549640 1 Processor -- start 2024-04-06T02:49:11.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.160+0000 7f3e92549640 1 -- start start 2024-04-06T02:49:11.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.161+0000 7f3e92549640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e8c100b80 0x7f3e8c1004f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:11.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.161+0000 7f3e92549640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e8c1034b0 0x7f3e8c0feb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:11.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.161+0000 7f3e92549640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c0ff070 con 0x7f3e8c1034b0 2024-04-06T02:49:11.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.161+0000 7f3e92549640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c0ff1e0 con 0x7f3e8c100b80 2024-04-06T02:49:11.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.161+0000 7f3e8b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e8c1034b0 0x7f3e8c0feb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:11.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.161+0000 7f3e8b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e8c1034b0 0x7f3e8c0feb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:51002/0 (socket says 172.21.15.105:51002) 2024-04-06T02:49:11.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.161+0000 7f3e8b7fe640 1 -- 172.21.15.105:0/733394126 learned_addr learned my addr 172.21.15.105:0/733394126 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:49:11.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.161+0000 7f3e8bfff640 1 --2- 172.21.15.105:0/733394126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e8c100b80 0x7f3e8c1004f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:11.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.161+0000 7f3e8b7fe640 1 -- 172.21.15.105:0/733394126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e8c100b80 msgr2=0x7f3e8c1004f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:11.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.161+0000 7f3e8b7fe640 1 --2- 172.21.15.105:0/733394126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e8c100b80 0x7f3e8c1004f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:11.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.161+0000 7f3e8b7fe640 1 -- 172.21.15.105:0/733394126 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e80009560 con 0x7f3e8c1034b0 2024-04-06T02:49:11.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.161+0000 7f3e8b7fe640 1 --2- 172.21.15.105:0/733394126 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e8c1034b0 0x7f3e8c0feb30 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c00e970 tx=0x7f3e7c00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:11.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.161+0000 7f3e897fa640 1 -- 172.21.15.105:0/733394126 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e7c00cc90 con 0x7f3e8c1034b0 2024-04-06T02:49:11.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.161+0000 7f3e897fa640 1 -- 172.21.15.105:0/733394126 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3e7c00cdf0 con 0x7f3e8c1034b0 2024-04-06T02:49:11.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.161+0000 7f3e92549640 1 -- 172.21.15.105:0/733394126 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e8c0ff4c0 con 0x7f3e8c1034b0 2024-04-06T02:49:11.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.161+0000 7f3e897fa640 1 -- 172.21.15.105:0/733394126 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e7c010430 con 0x7f3e8c1034b0 2024-04-06T02:49:11.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.161+0000 7f3e92549640 1 -- 172.21.15.105:0/733394126 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e8c19d990 con 0x7f3e8c1034b0 2024-04-06T02:49:11.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.162+0000 7f3e92549640 1 -- 172.21.15.105:0/733394126 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e8c18f9d0 con 0x7f3e8c1034b0 2024-04-06T02:49:11.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.163+0000 7f3e897fa640 1 -- 172.21.15.105:0/733394126 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f3e7c0046c0 con 0x7f3e8c1034b0 2024-04-06T02:49:11.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.163+0000 7f3e897fa640 1 --2- 172.21.15.105:0/733394126 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f3e7406c450 0x7f3e7406e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:11.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.163+0000 7f3e897fa640 1 -- 172.21.15.105:0/733394126 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f3e7c014070 con 0x7f3e8c1034b0 2024-04-06T02:49:11.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.163+0000 7f3e8bfff640 1 --2- 172.21.15.105:0/733394126 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f3e7406c450 0x7f3e7406e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:11.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.163+0000 7f3e8bfff640 1 --2- 172.21.15.105:0/733394126 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f3e7406c450 0x7f3e7406e910 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3e80006010 tx=0x7f3e800058d0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:11.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.167+0000 7f3e897fa640 1 -- 172.21.15.105:0/733394126 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3e7c0564c0 con 0x7f3e8c1034b0 2024-04-06T02:49:11.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.336+0000 7f3e92549640 1 -- 172.21.15.105:0/733394126 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3e8c108e00 con 0x7f3e8c1034b0 2024-04-06T02:49:11.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.337+0000 7f3e897fa640 1 -- 172.21.15.105:0/733394126 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3e7c059ae0 con 0x7f3e8c1034b0 2024-04-06T02:49:11.322 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 1 2024-04-06T02:49:11.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.338+0000 7f3e92549640 1 -- 172.21.15.105:0/733394126 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f3e7406c450 msgr2=0x7f3e7406e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:11.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.338+0000 7f3e92549640 1 --2- 172.21.15.105:0/733394126 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f3e7406c450 0x7f3e7406e910 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3e80006010 tx=0x7f3e800058d0 comp rx=0 tx=0).stop 2024-04-06T02:49:11.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.338+0000 7f3e92549640 1 -- 172.21.15.105:0/733394126 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e8c1034b0 msgr2=0x7f3e8c0feb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:11.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.338+0000 7f3e92549640 1 --2- 172.21.15.105:0/733394126 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e8c1034b0 0x7f3e8c0feb30 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c00e970 tx=0x7f3e7c00ee30 comp rx=0 tx=0).stop 2024-04-06T02:49:11.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.339+0000 7f3e92549640 1 -- 172.21.15.105:0/733394126 shutdown_connections 2024-04-06T02:49:11.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.339+0000 7f3e92549640 1 --2- 172.21.15.105:0/733394126 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e8c1034b0 0x7f3e8c0feb30 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:11.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.339+0000 7f3e92549640 1 --2- 172.21.15.105:0/733394126 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f3e7406c450 0x7f3e7406e910 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:11.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.339+0000 7f3e92549640 1 --2- 172.21.15.105:0/733394126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3e8c100b80 0x7f3e8c1004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:11.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.339+0000 7f3e92549640 1 -- 172.21.15.105:0/733394126 >> 172.21.15.105:0/733394126 conn(0x7f3e8c0fa530 msgr2=0x7f3e8c0fc920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:49:11.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.339+0000 7f3e92549640 1 -- 172.21.15.105:0/733394126 shutdown_connections 2024-04-06T02:49:11.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.339+0000 7f3e92549640 1 -- 172.21.15.105:0/733394126 wait complete. 2024-04-06T02:49:11.336 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-06T02:49:11.445 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.459+0000 7f591c605640 1 -- 172.21.15.105:0/4142596181 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59141000d0 msgr2=0x7f59141004f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:11.445 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.459+0000 7f591c605640 1 --2- 172.21.15.105:0/4142596181 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59141000d0 0x7f59141004f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5910009880 tx=0x7f591002bce0 comp rx=0 tx=0).stop 2024-04-06T02:49:11.447 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.460+0000 7f591c605640 1 -- 172.21.15.105:0/4142596181 shutdown_connections 2024-04-06T02:49:11.447 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.460+0000 7f591c605640 1 --2- 172.21.15.105:0/4142596181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59141012c0 0x7f5914101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:11.447 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.460+0000 7f591c605640 1 --2- 172.21.15.105:0/4142596181 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59141000d0 0x7f59141004f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:11.447 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.460+0000 7f591c605640 1 -- 172.21.15.105:0/4142596181 >> 172.21.15.105:0/4142596181 conn(0x7f59140fb680 msgr2=0x7f59140fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:49:11.447 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.460+0000 7f591c605640 1 -- 172.21.15.105:0/4142596181 shutdown_connections 2024-04-06T02:49:11.447 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.460+0000 7f591c605640 1 -- 172.21.15.105:0/4142596181 wait complete. 2024-04-06T02:49:11.448 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.460+0000 7f591c605640 1 Processor -- start 2024-04-06T02:49:11.448 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.460+0000 7f591c605640 1 -- start start 2024-04-06T02:49:11.448 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.461+0000 7f591c605640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59141000d0 0x7f5914195810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:11.448 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.461+0000 7f591c605640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59141012c0 0x7f5914195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:11.448 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.461+0000 7f591c605640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5914196320 con 0x7f59141000d0 2024-04-06T02:49:11.448 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.461+0000 7f591c605640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5914196490 con 0x7f59141012c0 2024-04-06T02:49:11.448 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.461+0000 7f591a3a1640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59141000d0 0x7f5914195810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:11.448 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.461+0000 7f591a3a1640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59141000d0 0x7f5914195810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:51018/0 (socket says 172.21.15.105:51018) 2024-04-06T02:49:11.448 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.461+0000 7f591a3a1640 1 -- 172.21.15.105:0/1725362754 learned_addr learned my addr 172.21.15.105:0/1725362754 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:49:11.448 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.461+0000 7f5919ba0640 1 --2- 172.21.15.105:0/1725362754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59141012c0 0x7f5914195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:11.448 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.461+0000 7f591a3a1640 1 -- 172.21.15.105:0/1725362754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59141012c0 msgr2=0x7f5914195d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:11.448 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.461+0000 7f591a3a1640 1 --2- 172.21.15.105:0/1725362754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59141012c0 0x7f5914195d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:11.448 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.461+0000 7f591a3a1640 1 -- 172.21.15.105:0/1725362754 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5910009560 con 0x7f59141000d0 2024-04-06T02:49:11.448 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.461+0000 7f591a3a1640 1 --2- 172.21.15.105:0/1725362754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59141000d0 0x7f5914195810 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f591002c220 tx=0x7f5910004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:11.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.461+0000 7f590b7fe640 1 -- 172.21.15.105:0/1725362754 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59100051d0 con 0x7f59141000d0 2024-04-06T02:49:11.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.461+0000 7f591c605640 1 -- 172.21.15.105:0/1725362754 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f591419aec0 con 0x7f59141000d0 2024-04-06T02:49:11.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.461+0000 7f590b7fe640 1 -- 172.21.15.105:0/1725362754 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f59100035f0 con 0x7f59141000d0 2024-04-06T02:49:11.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.461+0000 7f590b7fe640 1 -- 172.21.15.105:0/1725362754 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5910034710 con 0x7f59141000d0 2024-04-06T02:49:11.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.462+0000 7f591c605640 1 -- 172.21.15.105:0/1725362754 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f591419b3c0 con 0x7f59141000d0 2024-04-06T02:49:11.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.463+0000 7f591c605640 1 -- 172.21.15.105:0/1725362754 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5914108ec0 con 0x7f59141000d0 2024-04-06T02:49:11.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.463+0000 7f590b7fe640 1 -- 172.21.15.105:0/1725362754 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f5910003760 con 0x7f59141000d0 2024-04-06T02:49:11.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.463+0000 7f590b7fe640 1 --2- 172.21.15.105:0/1725362754 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f590006c450 0x7f590006e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:11.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.463+0000 7f590b7fe640 1 -- 172.21.15.105:0/1725362754 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f59100ae2a0 con 0x7f59141000d0 2024-04-06T02:49:11.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.463+0000 7f5919ba0640 1 --2- 172.21.15.105:0/1725362754 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f590006c450 0x7f590006e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:11.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.464+0000 7f5919ba0640 1 --2- 172.21.15.105:0/1725362754 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f590006c450 0x7f590006e910 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5914196d20 tx=0x7f5904009110 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:11.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.468+0000 7f590b7fe640 1 -- 172.21.15.105:0/1725362754 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5910078a20 con 0x7f59141000d0 2024-04-06T02:49:11.628 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.643+0000 7f591c605640 1 -- 172.21.15.105:0/1725362754 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f591419b630 con 0x7f59141000d0 2024-04-06T02:49:11.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.644+0000 7f590b7fe640 1 -- 172.21.15.105:0/1725362754 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7f591007c040 con 0x7f59141000d0 2024-04-06T02:49:11.629 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 32 2024-04-06T02:49:11.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.645+0000 7f591c605640 1 -- 172.21.15.105:0/1725362754 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f590006c450 msgr2=0x7f590006e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:11.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.645+0000 7f591c605640 1 --2- 172.21.15.105:0/1725362754 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f590006c450 0x7f590006e910 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5914196d20 tx=0x7f5904009110 comp rx=0 tx=0).stop 2024-04-06T02:49:11.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.645+0000 7f591c605640 1 -- 172.21.15.105:0/1725362754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59141000d0 msgr2=0x7f5914195810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:11.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.645+0000 7f591c605640 1 --2- 172.21.15.105:0/1725362754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59141000d0 0x7f5914195810 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f591002c220 tx=0x7f5910004270 comp rx=0 tx=0).stop 2024-04-06T02:49:11.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.645+0000 7f591c605640 1 -- 172.21.15.105:0/1725362754 shutdown_connections 2024-04-06T02:49:11.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.645+0000 7f591c605640 1 --2- 172.21.15.105:0/1725362754 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f590006c450 0x7f590006e910 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:11.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.645+0000 7f591c605640 1 --2- 172.21.15.105:0/1725362754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59141012c0 0x7f5914195d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:11.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.645+0000 7f591c605640 1 --2- 172.21.15.105:0/1725362754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59141000d0 0x7f5914195810 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:11.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.645+0000 7f591c605640 1 -- 172.21.15.105:0/1725362754 >> 172.21.15.105:0/1725362754 conn(0x7f59140fb680 msgr2=0x7f59140fd9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:49:11.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.646+0000 7f591c605640 1 -- 172.21.15.105:0/1725362754 shutdown_connections 2024-04-06T02:49:11.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.646+0000 7f591c605640 1 -- 172.21.15.105:0/1725362754 wait complete. 2024-04-06T02:49:11.642 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-06T02:49:11.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.762+0000 7f2be2a45640 1 -- 172.21.15.105:0/2549303447 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2bdc1000d0 msgr2=0x7f2bdc1004f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:11.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.762+0000 7f2be2a45640 1 --2- 172.21.15.105:0/2549303447 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2bdc1000d0 0x7f2bdc1004f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc009880 tx=0x7f2bcc02bce0 comp rx=0 tx=0).stop 2024-04-06T02:49:11.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.762+0000 7f2be2a45640 1 -- 172.21.15.105:0/2549303447 shutdown_connections 2024-04-06T02:49:11.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.762+0000 7f2be2a45640 1 --2- 172.21.15.105:0/2549303447 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bdc1012c0 0x7f2bdc101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:11.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.762+0000 7f2be2a45640 1 --2- 172.21.15.105:0/2549303447 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2bdc1000d0 0x7f2bdc1004f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:11.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.762+0000 7f2be2a45640 1 -- 172.21.15.105:0/2549303447 >> 172.21.15.105:0/2549303447 conn(0x7f2bdc0fb620 msgr2=0x7f2bdc0fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:49:11.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.762+0000 7f2be2a45640 1 -- 172.21.15.105:0/2549303447 shutdown_connections 2024-04-06T02:49:11.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.762+0000 7f2be2a45640 1 -- 172.21.15.105:0/2549303447 wait complete. 2024-04-06T02:49:11.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.762+0000 7f2be2a45640 1 Processor -- start 2024-04-06T02:49:11.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.762+0000 7f2be2a45640 1 -- start start 2024-04-06T02:49:11.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.763+0000 7f2be2a45640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2bdc1000d0 0x7f2bdc1959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:11.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.763+0000 7f2be2a45640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bdc1012c0 0x7f2bdc195f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:11.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.763+0000 7f2be2a45640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc1964d0 con 0x7f2bdc1000d0 2024-04-06T02:49:11.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.763+0000 7f2be2a45640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc196640 con 0x7f2bdc1012c0 2024-04-06T02:49:11.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.763+0000 7f2bdbfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2bdc1000d0 0x7f2bdc1959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:11.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.763+0000 7f2bdbfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2bdc1000d0 0x7f2bdc1959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:51038/0 (socket says 172.21.15.105:51038) 2024-04-06T02:49:11.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.763+0000 7f2bdbfff640 1 -- 172.21.15.105:0/3126388266 learned_addr learned my addr 172.21.15.105:0/3126388266 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:49:11.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.763+0000 7f2bdb7fe640 1 --2- 172.21.15.105:0/3126388266 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bdc1012c0 0x7f2bdc195f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:11.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.763+0000 7f2bdbfff640 1 -- 172.21.15.105:0/3126388266 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bdc1012c0 msgr2=0x7f2bdc195f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:11.751 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.763+0000 7f2bdbfff640 1 --2- 172.21.15.105:0/3126388266 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bdc1012c0 0x7f2bdc195f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:11.751 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.763+0000 7f2bdbfff640 1 -- 172.21.15.105:0/3126388266 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bcc009560 con 0x7f2bdc1000d0 2024-04-06T02:49:11.751 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.763+0000 7f2bdbfff640 1 --2- 172.21.15.105:0/3126388266 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2bdc1000d0 0x7f2bdc1959c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc000c00 tx=0x7f2bcc004f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:11.751 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.763+0000 7f2bd97fa640 1 -- 172.21.15.105:0/3126388266 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bcc004520 con 0x7f2bdc1000d0 2024-04-06T02:49:11.751 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.763+0000 7f2be2a45640 1 -- 172.21.15.105:0/3126388266 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bdc19b070 con 0x7f2bdc1000d0 2024-04-06T02:49:11.751 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.763+0000 7f2bd97fa640 1 -- 172.21.15.105:0/3126388266 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2bcc0054b0 con 0x7f2bdc1000d0 2024-04-06T02:49:11.751 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.763+0000 7f2bd97fa640 1 -- 172.21.15.105:0/3126388266 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bcc034440 con 0x7f2bdc1000d0 2024-04-06T02:49:11.751 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.763+0000 7f2be2a45640 1 -- 172.21.15.105:0/3126388266 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bdc19b5f0 con 0x7f2bdc1000d0 2024-04-06T02:49:11.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.764+0000 7f2be2a45640 1 -- 172.21.15.105:0/3126388266 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bdc108ec0 con 0x7f2bdc1000d0 2024-04-06T02:49:11.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.765+0000 7f2bd97fa640 1 -- 172.21.15.105:0/3126388266 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f2bcc005000 con 0x7f2bdc1000d0 2024-04-06T02:49:11.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.765+0000 7f2bd97fa640 1 --2- 172.21.15.105:0/3126388266 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f2bc406c450 0x7f2bc406e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:11.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.765+0000 7f2bd97fa640 1 -- 172.21.15.105:0/3126388266 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f2bcc0adf10 con 0x7f2bdc1000d0 2024-04-06T02:49:11.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.765+0000 7f2bdb7fe640 1 --2- 172.21.15.105:0/3126388266 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f2bc406c450 0x7f2bc406e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:11.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.765+0000 7f2bdb7fe640 1 --2- 172.21.15.105:0/3126388266 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f2bc406c450 0x7f2bc406e910 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc196ed0 tx=0x7f2bd0008040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:11.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.769+0000 7f2bd97fa640 1 -- 172.21.15.105:0/3126388266 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2bcc078690 con 0x7f2bdc1000d0 2024-04-06T02:49:11.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:11 smithi105 ceph-mon[29228]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-06T02:49:11.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:11 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/2402301678' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T02:49:11.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:11 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/733394126' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-06T02:49:11.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:11 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/1725362754' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-06T02:49:11.932 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.947+0000 7f2be2a45640 1 -- 172.21.15.105:0/3126388266 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f2bdc19ba10 con 0x7f2bdc1000d0 2024-04-06T02:49:11.933 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.947+0000 7f2bd97fa640 1 -- 172.21.15.105:0/3126388266 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7f2bcc07bcb0 con 0x7f2bdc1000d0 2024-04-06T02:49:11.933 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 32 2024-04-06T02:49:11.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.949+0000 7f2be2a45640 1 -- 172.21.15.105:0/3126388266 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f2bc406c450 msgr2=0x7f2bc406e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:11.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.949+0000 7f2be2a45640 1 --2- 172.21.15.105:0/3126388266 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f2bc406c450 0x7f2bc406e910 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc196ed0 tx=0x7f2bd0008040 comp rx=0 tx=0).stop 2024-04-06T02:49:11.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.949+0000 7f2be2a45640 1 -- 172.21.15.105:0/3126388266 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2bdc1000d0 msgr2=0x7f2bdc1959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:11.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.949+0000 7f2be2a45640 1 --2- 172.21.15.105:0/3126388266 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2bdc1000d0 0x7f2bdc1959c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc000c00 tx=0x7f2bcc004f00 comp rx=0 tx=0).stop 2024-04-06T02:49:11.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.949+0000 7f2be2a45640 1 -- 172.21.15.105:0/3126388266 shutdown_connections 2024-04-06T02:49:11.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.949+0000 7f2be2a45640 1 --2- 172.21.15.105:0/3126388266 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f2bc406c450 0x7f2bc406e910 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:11.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.949+0000 7f2be2a45640 1 --2- 172.21.15.105:0/3126388266 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bdc1012c0 0x7f2bdc195f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:11.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.949+0000 7f2be2a45640 1 --2- 172.21.15.105:0/3126388266 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2bdc1000d0 0x7f2bdc1959c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:11.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.949+0000 7f2be2a45640 1 -- 172.21.15.105:0/3126388266 >> 172.21.15.105:0/3126388266 conn(0x7f2bdc0fb620 msgr2=0x7f2bdc1044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:49:11.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.949+0000 7f2be2a45640 1 -- 172.21.15.105:0/3126388266 shutdown_connections 2024-04-06T02:49:11.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:11.949+0000 7f2be2a45640 1 -- 172.21.15.105:0/3126388266 wait complete. 2024-04-06T02:49:11.946 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-06T02:49:11.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:11 smithi184 ceph-mon[37439]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-06T02:49:11.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:11 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/2402301678' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T02:49:11.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:11 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/733394126' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-06T02:49:11.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:11 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/1725362754' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-06T02:49:12.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.062+0000 7f2a429fb640 1 -- 172.21.15.105:0/3547720662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2a3c1000d0 msgr2=0x7f2a3c1004f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:12.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.062+0000 7f2a429fb640 1 --2- 172.21.15.105:0/3547720662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2a3c1000d0 0x7f2a3c1004f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c009880 tx=0x7f2a2c02bce0 comp rx=0 tx=0).stop 2024-04-06T02:49:12.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.062+0000 7f2a429fb640 1 -- 172.21.15.105:0/3547720662 shutdown_connections 2024-04-06T02:49:12.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.062+0000 7f2a429fb640 1 --2- 172.21.15.105:0/3547720662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a3c1012c0 0x7f2a3c101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:12.050 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.062+0000 7f2a429fb640 1 --2- 172.21.15.105:0/3547720662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2a3c1000d0 0x7f2a3c1004f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:12.050 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.062+0000 7f2a429fb640 1 -- 172.21.15.105:0/3547720662 >> 172.21.15.105:0/3547720662 conn(0x7f2a3c0fb640 msgr2=0x7f2a3c0fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:49:12.050 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.062+0000 7f2a429fb640 1 -- 172.21.15.105:0/3547720662 shutdown_connections 2024-04-06T02:49:12.050 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.062+0000 7f2a429fb640 1 -- 172.21.15.105:0/3547720662 wait complete. 2024-04-06T02:49:12.050 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.062+0000 7f2a429fb640 1 Processor -- start 2024-04-06T02:49:12.050 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.062+0000 7f2a429fb640 1 -- start start 2024-04-06T02:49:12.050 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.063+0000 7f2a429fb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a3c1000d0 0x7f2a3c1958b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:12.050 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.063+0000 7f2a429fb640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2a3c1012c0 0x7f2a3c195df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:12.050 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.063+0000 7f2a429fb640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c1963c0 con 0x7f2a3c1012c0 2024-04-06T02:49:12.050 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.063+0000 7f2a429fb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c196530 con 0x7f2a3c1000d0 2024-04-06T02:49:12.050 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.063+0000 7f2a3b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2a3c1012c0 0x7f2a3c195df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:12.050 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.063+0000 7f2a3b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2a3c1012c0 0x7f2a3c195df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:51058/0 (socket says 172.21.15.105:51058) 2024-04-06T02:49:12.050 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.063+0000 7f2a3b7fe640 1 -- 172.21.15.105:0/4169713718 learned_addr learned my addr 172.21.15.105:0/4169713718 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:49:12.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.063+0000 7f2a3bfff640 1 --2- 172.21.15.105:0/4169713718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a3c1000d0 0x7f2a3c1958b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:12.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.063+0000 7f2a3b7fe640 1 -- 172.21.15.105:0/4169713718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a3c1000d0 msgr2=0x7f2a3c1958b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:12.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.063+0000 7f2a3b7fe640 1 --2- 172.21.15.105:0/4169713718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a3c1000d0 0x7f2a3c1958b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:12.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.063+0000 7f2a3b7fe640 1 -- 172.21.15.105:0/4169713718 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a2c009560 con 0x7f2a3c1012c0 2024-04-06T02:49:12.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.063+0000 7f2a3b7fe640 1 --2- 172.21.15.105:0/4169713718 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2a3c1012c0 0x7f2a3c195df0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2a3000d6f0 tx=0x7f2a3000dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:12.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.063+0000 7f2a397fa640 1 -- 172.21.15.105:0/4169713718 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a30009a30 con 0x7f2a3c1012c0 2024-04-06T02:49:12.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.064+0000 7f2a397fa640 1 -- 172.21.15.105:0/4169713718 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2a300054d0 con 0x7f2a3c1012c0 2024-04-06T02:49:12.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.064+0000 7f2a429fb640 1 -- 172.21.15.105:0/4169713718 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a3c19afc0 con 0x7f2a3c1012c0 2024-04-06T02:49:12.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.064+0000 7f2a397fa640 1 -- 172.21.15.105:0/4169713718 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a30010460 con 0x7f2a3c1012c0 2024-04-06T02:49:12.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.064+0000 7f2a429fb640 1 -- 172.21.15.105:0/4169713718 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a3c19b4f0 con 0x7f2a3c1012c0 2024-04-06T02:49:12.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.064+0000 7f2a429fb640 1 -- 172.21.15.105:0/4169713718 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a3c108e60 con 0x7f2a3c1012c0 2024-04-06T02:49:12.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.065+0000 7f2a397fa640 1 -- 172.21.15.105:0/4169713718 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f2a30009b90 con 0x7f2a3c1012c0 2024-04-06T02:49:12.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.065+0000 7f2a397fa640 1 --2- 172.21.15.105:0/4169713718 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f2a2406c510 0x7f2a2406e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:12.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.065+0000 7f2a397fa640 1 -- 172.21.15.105:0/4169713718 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f2a3008b370 con 0x7f2a3c1012c0 2024-04-06T02:49:12.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.065+0000 7f2a3bfff640 1 --2- 172.21.15.105:0/4169713718 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f2a2406c510 0x7f2a2406e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:12.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.066+0000 7f2a3bfff640 1 --2- 172.21.15.105:0/4169713718 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f2a2406c510 0x7f2a2406e9d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c000c00 tx=0x7f2a2c005fd0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:12.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.070+0000 7f2a397fa640 1 -- 172.21.15.105:0/4169713718 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2a30055ba0 con 0x7f2a3c1012c0 2024-04-06T02:49:12.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.283+0000 7f2a429fb640 1 -- 172.21.15.105:0/4169713718 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a3c19b710 con 0x7f2a3c1012c0 2024-04-06T02:49:12.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.284+0000 7f2a397fa640 1 -- 172.21.15.105:0/4169713718 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4763 (secure 0 0 0) 0x7f2a300591c0 con 0x7f2a3c1012c0 2024-04-06T02:49:12.270 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-06T02:49:12.270 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14496,"name":"cephfs.smithi105.mrivth","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6827/2109838618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":2109838618},{"type":"v1","addr":"172.21.15.105:6827","nonce":2109838618}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24265,"name":"cephfs.smithi184.htgrjn","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.184:6827/323195824","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":323195824},{"type":"v1","addr":"172.21.15.184:6827","nonce":323195824}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-06T02:48:41.709017+0000","modified":"2024-04-06T02:49:02.531386+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14486,"mds_1":24259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14486":{"gid":14486,"name":"cephfs.smithi105.oxxcbq","rank":0,"incarnation":11,"state":"up:active","state_seq":6,"addr":"172.21.15.105:6829/2512987070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":2512987070},{"type":"v1","addr":"172.21.15.105:6829","nonce":2512987070}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24259":{"gid":24259,"name":"cephfs.smithi184.oochxp","rank":1,"incarnation":6,"state":"up:active","state_seq":4,"addr":"172.21.15.184:6825/2770403500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2770403500},{"type":"v1","addr":"172.21.15.184:6825","nonce":2770403500}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-06T02:49:12.270 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 15 2024-04-06T02:49:12.272 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.286+0000 7f2a429fb640 1 -- 172.21.15.105:0/4169713718 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f2a2406c510 msgr2=0x7f2a2406e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:12.272 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.286+0000 7f2a429fb640 1 --2- 172.21.15.105:0/4169713718 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f2a2406c510 0x7f2a2406e9d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c000c00 tx=0x7f2a2c005fd0 comp rx=0 tx=0).stop 2024-04-06T02:49:12.272 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.286+0000 7f2a429fb640 1 -- 172.21.15.105:0/4169713718 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2a3c1012c0 msgr2=0x7f2a3c195df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:12.272 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.286+0000 7f2a429fb640 1 --2- 172.21.15.105:0/4169713718 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2a3c1012c0 0x7f2a3c195df0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2a3000d6f0 tx=0x7f2a3000dbb0 comp rx=0 tx=0).stop 2024-04-06T02:49:12.272 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.286+0000 7f2a429fb640 1 -- 172.21.15.105:0/4169713718 shutdown_connections 2024-04-06T02:49:12.272 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.286+0000 7f2a429fb640 1 --2- 172.21.15.105:0/4169713718 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2a3c1012c0 0x7f2a3c195df0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:12.272 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.286+0000 7f2a429fb640 1 --2- 172.21.15.105:0/4169713718 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f2a2406c510 0x7f2a2406e9d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:12.272 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.286+0000 7f2a429fb640 1 --2- 172.21.15.105:0/4169713718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a3c1000d0 0x7f2a3c1958b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:12.272 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.286+0000 7f2a429fb640 1 -- 172.21.15.105:0/4169713718 >> 172.21.15.105:0/4169713718 conn(0x7f2a3c0fb640 msgr2=0x7f2a3c1044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:49:12.272 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.286+0000 7f2a429fb640 1 -- 172.21.15.105:0/4169713718 shutdown_connections 2024-04-06T02:49:12.272 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:12.286+0000 7f2a429fb640 1 -- 172.21.15.105:0/4169713718 wait complete. 2024-04-06T02:49:12.283 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 15, 'max_mds': 2, 'flags': 18} 2024-04-06T02:49:12.283 INFO:teuthology.run_tasks:Running task kclient... 2024-04-06T02:49:12.306 INFO:tasks.kclient:Mounting kernel clients... 2024-04-06T02:49:12.307 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-04-06T02:49:12.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T02:49:12.307 DEBUG:teuthology.orchestra.run.smithi105:> ip netns list 2024-04-06T02:49:12.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T02:49:12.322 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip link delete ceph-brx 2024-04-06T02:49:12.390 INFO:teuthology.orchestra.run.smithi105.stderr:Cannot find device "ceph-brx" 2024-04-06T02:49:12.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T02:49:12.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T02:49:12.391 DEBUG:teuthology.orchestra.run.smithi184:> ip netns list 2024-04-06T02:49:12.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T02:49:12.411 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link delete ceph-brx 2024-04-06T02:49:12.488 INFO:teuthology.orchestra.run.smithi184.stderr:Cannot find device "ceph-brx" 2024-04-06T02:49:12.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T02:49:12.490 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-04-06T02:49:12.490 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-06T02:49:12.490 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-06T02:49:12.490 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph fs ls 2024-04-06T02:49:12.730 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:49:12.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:12 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/3126388266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-06T02:49:12.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:12 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/4169713718' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-06T02:49:12.852 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:12 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/3126388266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-06T02:49:12.852 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:12 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/4169713718' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-06T02:49:14.112 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:13 smithi105 ceph-mon[29228]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-06T02:49:14.112 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:13 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:49:14.112 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:13 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:49:14.112 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:13 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:49:14.112 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:13 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:49:14.112 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:13 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:49:14.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:13 smithi184 ceph-mon[37439]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-06T02:49:14.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:13 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:49:14.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:13 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:49:14.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:13 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:49:14.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:13 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:49:14.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:13 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:49:14.478 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.490+0000 7f642a205700 1 -- 172.21.15.105:0/2186435045 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64240ffa50 msgr2=0x7f64240ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:14.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.490+0000 7f642a205700 1 --2- 172.21.15.105:0/2186435045 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64240ffa50 0x7f64240ffed0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6418009910 tx=0x7f6418009c20 comp rx=0 tx=0).stop 2024-04-06T02:49:14.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.490+0000 7f642a205700 1 -- 172.21.15.105:0/2186435045 shutdown_connections 2024-04-06T02:49:14.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.490+0000 7f642a205700 1 --2- 172.21.15.105:0/2186435045 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64240ffa50 0x7f64240ffed0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:14.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.490+0000 7f642a205700 1 --2- 172.21.15.105:0/2186435045 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6424105a50 0x7f6424105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:14.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.490+0000 7f642a205700 1 -- 172.21.15.105:0/2186435045 >> 172.21.15.105:0/2186435045 conn(0x7f64240fb560 msgr2=0x7f64240fd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:49:14.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.490+0000 7f642a205700 1 -- 172.21.15.105:0/2186435045 shutdown_connections 2024-04-06T02:49:14.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.490+0000 7f642a205700 1 -- 172.21.15.105:0/2186435045 wait complete. 2024-04-06T02:49:14.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.491+0000 7f642a205700 1 Processor -- start 2024-04-06T02:49:14.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.491+0000 7f642a205700 1 -- start start 2024-04-06T02:49:14.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.491+0000 7f642a205700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64240ffa50 0x7f642419c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:14.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.491+0000 7f642a205700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6424105a50 0x7f642419cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:14.480 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.491+0000 7f642a205700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6424196590 con 0x7f64240ffa50 2024-04-06T02:49:14.480 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.491+0000 7f642a205700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6424196700 con 0x7f6424105a50 2024-04-06T02:49:14.480 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.491+0000 7f64237fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64240ffa50 0x7f642419c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:14.480 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.491+0000 7f64237fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64240ffa50 0x7f642419c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:51076/0 (socket says 172.21.15.105:51076) 2024-04-06T02:49:14.480 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.491+0000 7f64237fe700 1 -- 172.21.15.105:0/695953128 learned_addr learned my addr 172.21.15.105:0/695953128 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:49:14.480 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.492+0000 7f64237fe700 1 -- 172.21.15.105:0/695953128 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6424105a50 msgr2=0x7f642419cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:14.480 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.492+0000 7f64237fe700 1 --2- 172.21.15.105:0/695953128 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6424105a50 0x7f642419cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:14.480 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.492+0000 7f64237fe700 1 -- 172.21.15.105:0/695953128 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64180095f0 con 0x7f64240ffa50 2024-04-06T02:49:14.480 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.492+0000 7f64237fe700 1 --2- 172.21.15.105:0/695953128 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64240ffa50 0x7f642419c5a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f641400d8d0 tx=0x7f641400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:14.480 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.492+0000 7f6420ff9700 1 -- 172.21.15.105:0/695953128 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6414009940 con 0x7f64240ffa50 2024-04-06T02:49:14.480 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.492+0000 7f6420ff9700 1 -- 172.21.15.105:0/695953128 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6414010460 con 0x7f64240ffa50 2024-04-06T02:49:14.481 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.492+0000 7f642a205700 1 -- 172.21.15.105:0/695953128 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64241969e0 con 0x7f64240ffa50 2024-04-06T02:49:14.481 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.492+0000 7f6420ff9700 1 -- 172.21.15.105:0/695953128 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f641400f5f0 con 0x7f64240ffa50 2024-04-06T02:49:14.481 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.492+0000 7f642a205700 1 -- 172.21.15.105:0/695953128 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6424196f30 con 0x7f64240ffa50 2024-04-06T02:49:14.482 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.493+0000 7f642a205700 1 -- 172.21.15.105:0/695953128 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6424108ff0 con 0x7f64240ffa50 2024-04-06T02:49:14.482 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.493+0000 7f6420ff9700 1 -- 172.21.15.105:0/695953128 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f64140105d0 con 0x7f64240ffa50 2024-04-06T02:49:14.482 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.494+0000 7f6420ff9700 1 --2- 172.21.15.105:0/695953128 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f640c06c4e0 0x7f640c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:14.482 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.494+0000 7f6420ff9700 1 -- 172.21.15.105:0/695953128 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f641408b890 con 0x7f64240ffa50 2024-04-06T02:49:14.482 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.494+0000 7f6422ffd700 1 --2- 172.21.15.105:0/695953128 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f640c06c4e0 0x7f640c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:14.482 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.494+0000 7f6422ffd700 1 --2- 172.21.15.105:0/695953128 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f640c06c4e0 0x7f640c06e9a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6418009fd0 tx=0x7f641800b560 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:14.485 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.498+0000 7f6420ff9700 1 -- 172.21.15.105:0/695953128 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6414056010 con 0x7f64240ffa50 2024-04-06T02:49:14.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.733+0000 7f642a205700 1 -- 172.21.15.105:0/695953128 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6424197150 con 0x7f64240ffa50 2024-04-06T02:49:14.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.733+0000 7f6420ff9700 1 -- 172.21.15.105:0/695953128 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+83 (secure 0 0 0) 0x7f6414059630 con 0x7f64240ffa50 2024-04-06T02:49:14.720 INFO:teuthology.orchestra.run.smithi105.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-06T02:49:14.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.735+0000 7f642a205700 1 -- 172.21.15.105:0/695953128 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f640c06c4e0 msgr2=0x7f640c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:14.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.735+0000 7f642a205700 1 --2- 172.21.15.105:0/695953128 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f640c06c4e0 0x7f640c06e9a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6418009fd0 tx=0x7f641800b560 comp rx=0 tx=0).stop 2024-04-06T02:49:14.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.735+0000 7f642a205700 1 -- 172.21.15.105:0/695953128 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64240ffa50 msgr2=0x7f642419c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:14.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.735+0000 7f642a205700 1 --2- 172.21.15.105:0/695953128 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64240ffa50 0x7f642419c5a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f641400d8d0 tx=0x7f641400dc90 comp rx=0 tx=0).stop 2024-04-06T02:49:14.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.735+0000 7f642a205700 1 -- 172.21.15.105:0/695953128 shutdown_connections 2024-04-06T02:49:14.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.735+0000 7f642a205700 1 --2- 172.21.15.105:0/695953128 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f640c06c4e0 0x7f640c06e9a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:14.723 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.735+0000 7f642a205700 1 --2- 172.21.15.105:0/695953128 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64240ffa50 0x7f642419c5a0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:14.723 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.735+0000 7f642a205700 1 --2- 172.21.15.105:0/695953128 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6424105a50 0x7f642419cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:14.723 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.735+0000 7f642a205700 1 -- 172.21.15.105:0/695953128 >> 172.21.15.105:0/695953128 conn(0x7f64240fb560 msgr2=0x7f6424069050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:49:14.723 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.735+0000 7f642a205700 1 -- 172.21.15.105:0/695953128 shutdown_connections 2024-04-06T02:49:14.723 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:14.735+0000 7f642a205700 1 -- 172.21.15.105:0/695953128 wait complete. 2024-04-06T02:49:15.030 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:14 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/695953128' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-06T02:49:15.132 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:14 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/695953128' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-06T02:49:15.336 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-06T02:49:15.336 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-06T02:49:15.336 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi105.front.sepia.ceph.com 2024-04-06T02:49:15.336 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-06T02:49:15.336 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-06T02:49:15.336 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-06T02:49:15.336 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-06T02:49:15.336 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-06T02:49:15.336 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-06T02:49:15.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T02:49:15.337 DEBUG:teuthology.orchestra.run.smithi105:> ip addr 2024-04-06T02:49:15.353 INFO:teuthology.orchestra.run.smithi105.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T02:49:15.353 INFO:teuthology.orchestra.run.smithi105.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T02:49:15.353 INFO:teuthology.orchestra.run.smithi105.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T02:49:15.353 INFO:teuthology.orchestra.run.smithi105.stdout: valid_lft forever preferred_lft forever 2024-04-06T02:49:15.353 INFO:teuthology.orchestra.run.smithi105.stdout: inet6 ::1/128 scope host 2024-04-06T02:49:15.353 INFO:teuthology.orchestra.run.smithi105.stdout: valid_lft forever preferred_lft forever 2024-04-06T02:49:15.354 INFO:teuthology.orchestra.run.smithi105.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T02:49:15.354 INFO:teuthology.orchestra.run.smithi105.stdout: link/ether 0c:c4:7a:d8:a4:b8 brd ff:ff:ff:ff:ff:ff 2024-04-06T02:49:15.354 INFO:teuthology.orchestra.run.smithi105.stdout: altname enp7s0f0 2024-04-06T02:49:15.354 INFO:teuthology.orchestra.run.smithi105.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T02:49:15.354 INFO:teuthology.orchestra.run.smithi105.stdout: link/ether 0c:c4:7a:d8:a4:b9 brd ff:ff:ff:ff:ff:ff 2024-04-06T02:49:15.354 INFO:teuthology.orchestra.run.smithi105.stdout: altname enp7s0f1 2024-04-06T02:49:15.354 INFO:teuthology.orchestra.run.smithi105.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T02:49:15.354 INFO:teuthology.orchestra.run.smithi105.stdout: link/ether 0c:c4:7a:8f:d4:28 brd ff:ff:ff:ff:ff:ff 2024-04-06T02:49:15.354 INFO:teuthology.orchestra.run.smithi105.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-06T02:49:15.354 INFO:teuthology.orchestra.run.smithi105.stdout: link/ether 0c:c4:7a:8f:d4:29 brd ff:ff:ff:ff:ff:ff 2024-04-06T02:49:15.354 INFO:teuthology.orchestra.run.smithi105.stdout: inet 172.21.15.105/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-06T02:49:15.355 INFO:teuthology.orchestra.run.smithi105.stdout: valid_lft 42243sec preferred_lft 42243sec 2024-04-06T02:49:15.355 INFO:teuthology.orchestra.run.smithi105.stdout: inet6 fe80::ec4:7aff:fe8f:d429/64 scope link 2024-04-06T02:49:15.355 INFO:teuthology.orchestra.run.smithi105.stdout: valid_lft forever preferred_lft forever 2024-04-06T02:49:15.355 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-06T02:49:15.355 DEBUG:teuthology.orchestra.run.smithi105:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-06T02:49:15.355 DEBUG:teuthology.orchestra.run.smithi105:> set -e 2024-04-06T02:49:15.355 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip link add name ceph-brx type bridge 2024-04-06T02:49:15.356 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip addr flush dev ceph-brx 2024-04-06T02:49:15.356 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip link set ceph-brx up 2024-04-06T02:49:15.356 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-06T02:49:15.356 DEBUG:teuthology.orchestra.run.smithi105:> ') 2024-04-06T02:49:15.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-06T02:49:15.556 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-06T02:49:15.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T02:49:15.563 DEBUG:teuthology.orchestra.run.smithi105:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-06T02:49:15.590 INFO:teuthology.orchestra.run.smithi105.stdout:1 2024-04-06T02:49:15.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T02:49:15.591 DEBUG:teuthology.orchestra.run.smithi105:> route 2024-04-06T02:49:15.663 INFO:teuthology.orchestra.run.smithi105.stdout:Kernel IP routing table 2024-04-06T02:49:15.663 INFO:teuthology.orchestra.run.smithi105.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-06T02:49:15.663 INFO:teuthology.orchestra.run.smithi105.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-06T02:49:15.663 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-06T02:49:15.663 INFO:teuthology.orchestra.run.smithi105.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-06T02:49:15.664 DEBUG:teuthology.orchestra.run.smithi105:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-06T02:49:15.664 DEBUG:teuthology.orchestra.run.smithi105:> set -e 2024-04-06T02:49:15.664 DEBUG:teuthology.orchestra.run.smithi105:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-06T02:49:15.664 DEBUG:teuthology.orchestra.run.smithi105:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-06T02:49:15.664 DEBUG:teuthology.orchestra.run.smithi105:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-06T02:49:15.664 DEBUG:teuthology.orchestra.run.smithi105:> ') 2024-04-06T02:49:15.751 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-06T02:49:15.876 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:15 smithi105 ceph-mon[29228]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-06T02:49:15.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-06T02:49:15.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T02:49:15.992 DEBUG:teuthology.orchestra.run.smithi105:> ip netns list 2024-04-06T02:49:16.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T02:49:16.007 DEBUG:teuthology.orchestra.run.smithi105:> ip netns list-id 2024-04-06T02:49:16.062 DEBUG:teuthology.orchestra.run.smithi105:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-06T02:49:16.062 DEBUG:teuthology.orchestra.run.smithi105:> set -e 2024-04-06T02:49:16.062 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-06T02:49:16.062 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-06T02:49:16.062 DEBUG:teuthology.orchestra.run.smithi105:> ') 2024-04-06T02:49:16.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-06T02:49:16.178 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-06T02:49:16.185 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-06T02:49:16.185 DEBUG:teuthology.orchestra.run.smithi105:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-06T02:49:16.185 DEBUG:teuthology.orchestra.run.smithi105:> set -e 2024-04-06T02:49:16.185 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-06T02:49:16.185 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-06T02:49:16.185 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-06T02:49:16.185 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-06T02:49:16.185 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-06T02:49:16.186 DEBUG:teuthology.orchestra.run.smithi105:> ') 2024-04-06T02:49:16.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:15 smithi184 ceph-mon[37439]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-06T02:49:16.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-06T02:49:16.317 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-06T02:49:16.323 DEBUG:teuthology.orchestra.run.smithi105:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-06T02:49:16.323 DEBUG:teuthology.orchestra.run.smithi105:> set -e 2024-04-06T02:49:16.323 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip link set brx.0 up 2024-04-06T02:49:16.323 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip link set dev brx.0 master ceph-brx 2024-04-06T02:49:16.324 DEBUG:teuthology.orchestra.run.smithi105:> ') 2024-04-06T02:49:16.371 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-06T02:49:16.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-06T02:49:16.405 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T02:49:16.405 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-06T02:49:16.419 INFO:teuthology.orchestra.run.smithi105.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-06T02:49:16.420 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T02:49:16.420 DEBUG:teuthology.orchestra.run.smithi105:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-06T02:49:16.474 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-06T02:49:16.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T02:49:16.474 DEBUG:teuthology.orchestra.run.smithi105:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T02:49:16.575 INFO:teuthology.orchestra.run.smithi105.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T02:49:16.575 INFO:teuthology.orchestra.run.smithi105.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T02:49:16.575 INFO:teuthology.orchestra.run.smithi105.stdout:invalid new device string format 2024-04-06T02:49:16.687 INFO:teuthology.orchestra.run.smithi105.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T02:49:16.687 INFO:teuthology.orchestra.run.smithi105.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T02:49:16.687 INFO:teuthology.orchestra.run.smithi105.stdout:invalid new device string format 2024-04-06T02:49:16.687 INFO:teuthology.orchestra.run.smithi105.stdout:mount.ceph: resolved to: "172.21.15.105:3300,172.21.15.184:3300" 2024-04-06T02:49:16.687 INFO:teuthology.orchestra.run.smithi105.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T02:49:16.698 INFO:teuthology.orchestra.run.smithi105.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-06T02:49:16.698 INFO:teuthology.orchestra.run.smithi105.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-06T02:49:16.698 INFO:teuthology.orchestra.run.smithi105.stdout:invalid new device string format 2024-04-06T02:49:16.698 INFO:teuthology.orchestra.run.smithi105.stdout:mount.ceph: resolved to: "172.21.15.105:3300,172.21.15.184:3300" 2024-04-06T02:49:16.698 INFO:teuthology.orchestra.run.smithi105.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T02:49:16.698 INFO:teuthology.orchestra.run.smithi105.stdout:mount.ceph: resolved to: "172.21.15.105:6789,172.21.15.184:6789" 2024-04-06T02:49:16.698 INFO:teuthology.orchestra.run.smithi105.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.105:6789,172.21.15.184:6789:/ 2024-04-06T02:49:16.698 INFO:teuthology.orchestra.run.smithi105.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=130df578-f3bf-11ee-b649-cb9ed24678a4" will pass to kernel 2024-04-06T02:49:16.699 INFO:teuthology.orchestra.run.smithi105.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-06T02:49:16.699 INFO:teuthology.orchestra.run.smithi105.stdout: You just mounted a file system that supports labels which does not 2024-04-06T02:49:16.699 INFO:teuthology.orchestra.run.smithi105.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-06T02:49:16.699 INFO:teuthology.orchestra.run.smithi105.stdout: applications will generate AVC messages and not be allowed access to 2024-04-06T02:49:16.699 INFO:teuthology.orchestra.run.smithi105.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-06T02:49:16.701 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-06T02:49:16.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T02:49:16.701 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-06T02:49:16.731 DEBUG:teuthology.orchestra.run.smithi105:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T02:49:16.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:16 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-06T02:49:16.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-06T02:49:16.832 INFO:teuthology.orchestra.run.smithi105.stdout:client24297 2024-04-06T02:49:16.832 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-06T02:49:16.833 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T02:49:16.849 DEBUG:teuthology.orchestra.run.smithi105:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T02:49:16.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:16 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-06T02:49:16.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-06T02:49:16.949 INFO:teuthology.orchestra.run.smithi105.stdout:client24297 2024-04-06T02:49:16.949 DEBUG:teuthology.orchestra.run.smithi105:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T02:49:16.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:17 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-06T02:49:17.005 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-06T02:49:17.011 INFO:teuthology.orchestra.run.smithi105.stdout:client24297 2024-04-06T02:49:17.011 DEBUG:teuthology.orchestra.run.smithi105:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4.client24297/status') 2024-04-06T02:49:17.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-06T02:49:17.075 INFO:teuthology.orchestra.run.smithi105.stdout:instance: client.24297 (3)172.21.15.105:0/3238215151 2024-04-06T02:49:17.075 INFO:teuthology.orchestra.run.smithi105.stdout:blocklisted: false 2024-04-06T02:49:17.075 INFO:teuthology.orchestra.run.smithi105.stderr:0+1 records in 2024-04-06T02:49:17.075 INFO:teuthology.orchestra.run.smithi105.stderr:0+1 records out 2024-04-06T02:49:17.075 INFO:teuthology.orchestra.run.smithi105.stderr:72 bytes copied, 2.038e-05 s, 3.5 MB/s 2024-04-06T02:49:17.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-06T02:49:17.082 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-04-06T02:49:17.082 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-06T02:49:17.082 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-06T02:49:17.082 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -- ceph fs ls 2024-04-06T02:49:17.289 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:49:18.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:17 smithi105 ceph-mon[29228]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-06T02:49:18.237 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:17 smithi184 ceph-mon[37439]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-06T02:49:19.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.017+0000 7f7b9abb1700 1 -- 172.21.15.105:0/1768375368 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b94069020 msgr2=0x7f7b94105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:19.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.017+0000 7f7b9abb1700 1 --2- 172.21.15.105:0/1768375368 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b94069020 0x7f7b94105790 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7b84009910 tx=0x7f7b84009c20 comp rx=0 tx=0).stop 2024-04-06T02:49:19.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.017+0000 7f7b9abb1700 1 -- 172.21.15.105:0/1768375368 shutdown_connections 2024-04-06T02:49:19.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.017+0000 7f7b9abb1700 1 --2- 172.21.15.105:0/1768375368 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b94069020 0x7f7b94105790 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:19.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.017+0000 7f7b9abb1700 1 --2- 172.21.15.105:0/1768375368 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b94068670 0x7f7b94068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:19.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.017+0000 7f7b9abb1700 1 -- 172.21.15.105:0/1768375368 >> 172.21.15.105:0/1768375368 conn(0x7f7b940755a0 msgr2=0x7f7b940759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:49:19.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.017+0000 7f7b9abb1700 1 -- 172.21.15.105:0/1768375368 shutdown_connections 2024-04-06T02:49:19.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.017+0000 7f7b9abb1700 1 -- 172.21.15.105:0/1768375368 wait complete. 2024-04-06T02:49:19.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.017+0000 7f7b9abb1700 1 Processor -- start 2024-04-06T02:49:19.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.017+0000 7f7b9abb1700 1 -- start start 2024-04-06T02:49:19.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.018+0000 7f7b9abb1700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b94068670 0x7f7b9419e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:19.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.018+0000 7f7b9abb1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b94069020 0x7f7b9419ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:19.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.018+0000 7f7b9abb1700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9419f3a0 con 0x7f7b94069020 2024-04-06T02:49:19.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.018+0000 7f7b9abb1700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b94198850 con 0x7f7b94068670 2024-04-06T02:49:19.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.018+0000 7f7b8bfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b94069020 0x7f7b9419ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:19.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.018+0000 7f7b8bfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b94069020 0x7f7b9419ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:51092/0 (socket says 172.21.15.105:51092) 2024-04-06T02:49:19.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.018+0000 7f7b8bfff700 1 -- 172.21.15.105:0/2455319487 learned_addr learned my addr 172.21.15.105:0/2455319487 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:49:19.012 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.018+0000 7f7b9894d700 1 --2- 172.21.15.105:0/2455319487 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b94068670 0x7f7b9419e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:19.012 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.018+0000 7f7b8bfff700 1 -- 172.21.15.105:0/2455319487 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b94068670 msgr2=0x7f7b9419e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:19.012 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.018+0000 7f7b8bfff700 1 --2- 172.21.15.105:0/2455319487 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b94068670 0x7f7b9419e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:19.012 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.018+0000 7f7b8bfff700 1 -- 172.21.15.105:0/2455319487 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b840095f0 con 0x7f7b94069020 2024-04-06T02:49:19.012 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.018+0000 7f7b8bfff700 1 --2- 172.21.15.105:0/2455319487 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b94069020 0x7f7b9419ed10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7b84005f50 tx=0x7f7b8400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:19.012 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.018+0000 7f7b89ffb700 1 -- 172.21.15.105:0/2455319487 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b8401d070 con 0x7f7b94069020 2024-04-06T02:49:19.012 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.018+0000 7f7b89ffb700 1 -- 172.21.15.105:0/2455319487 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7b8400f460 con 0x7f7b94069020 2024-04-06T02:49:19.012 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.018+0000 7f7b9abb1700 1 -- 172.21.15.105:0/2455319487 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b94198ad0 con 0x7f7b94069020 2024-04-06T02:49:19.012 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.018+0000 7f7b89ffb700 1 -- 172.21.15.105:0/2455319487 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b84017630 con 0x7f7b94069020 2024-04-06T02:49:19.012 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.019+0000 7f7b9abb1700 1 -- 172.21.15.105:0/2455319487 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b94198fc0 con 0x7f7b94069020 2024-04-06T02:49:19.012 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.019+0000 7f7b9abb1700 1 -- 172.21.15.105:0/2455319487 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b9404ea90 con 0x7f7b94069020 2024-04-06T02:49:19.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.020+0000 7f7b89ffb700 1 -- 172.21.15.105:0/2455319487 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f7b8400f5d0 con 0x7f7b94069020 2024-04-06T02:49:19.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.020+0000 7f7b89ffb700 1 --2- 172.21.15.105:0/2455319487 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f7b7c06c4e0 0x7f7b7c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:19.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.021+0000 7f7b89ffb700 1 -- 172.21.15.105:0/2455319487 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f7b8408cd90 con 0x7f7b94069020 2024-04-06T02:49:19.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.021+0000 7f7b9894d700 1 --2- 172.21.15.105:0/2455319487 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f7b7c06c4e0 0x7f7b7c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:19.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.021+0000 7f7b9894d700 1 --2- 172.21.15.105:0/2455319487 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f7b7c06c4e0 0x7f7b7c06e9a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7b9000a400 tx=0x7f7b9000d040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:19.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.025+0000 7f7b89ffb700 1 -- 172.21.15.105:0/2455319487 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7b8405b020 con 0x7f7b94069020 2024-04-06T02:49:19.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.252+0000 7f7b9abb1700 1 -- 172.21.15.105:0/2455319487 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7b94199d60 con 0x7f7b94069020 2024-04-06T02:49:19.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.252+0000 7f7b89ffb700 1 -- 172.21.15.105:0/2455319487 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+83 (secure 0 0 0) 0x7f7b84055e60 con 0x7f7b94069020 2024-04-06T02:49:19.238 INFO:teuthology.orchestra.run.smithi105.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-06T02:49:19.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.254+0000 7f7b9abb1700 1 -- 172.21.15.105:0/2455319487 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f7b7c06c4e0 msgr2=0x7f7b7c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:19.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.254+0000 7f7b9abb1700 1 --2- 172.21.15.105:0/2455319487 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f7b7c06c4e0 0x7f7b7c06e9a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7b9000a400 tx=0x7f7b9000d040 comp rx=0 tx=0).stop 2024-04-06T02:49:19.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.254+0000 7f7b9abb1700 1 -- 172.21.15.105:0/2455319487 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b94069020 msgr2=0x7f7b9419ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:19.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.254+0000 7f7b9abb1700 1 --2- 172.21.15.105:0/2455319487 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b94069020 0x7f7b9419ed10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7b84005f50 tx=0x7f7b8400bd70 comp rx=0 tx=0).stop 2024-04-06T02:49:19.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.255+0000 7f7b9abb1700 1 -- 172.21.15.105:0/2455319487 shutdown_connections 2024-04-06T02:49:19.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.255+0000 7f7b9abb1700 1 --2- 172.21.15.105:0/2455319487 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f7b7c06c4e0 0x7f7b7c06e9a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:19.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.255+0000 7f7b9abb1700 1 --2- 172.21.15.105:0/2455319487 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b94068670 0x7f7b9419e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:19.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.255+0000 7f7b9abb1700 1 --2- 172.21.15.105:0/2455319487 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b94069020 0x7f7b9419ed10 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:19.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.255+0000 7f7b9abb1700 1 -- 172.21.15.105:0/2455319487 >> 172.21.15.105:0/2455319487 conn(0x7f7b940755a0 msgr2=0x7f7b94102760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:49:19.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.255+0000 7f7b9abb1700 1 -- 172.21.15.105:0/2455319487 shutdown_connections 2024-04-06T02:49:19.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:19.255+0000 7f7b9abb1700 1 -- 172.21.15.105:0/2455319487 wait complete. 2024-04-06T02:49:19.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:19 smithi105 ceph-mon[29228]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.0 KiB/s rd, 85 B/s wr, 3 op/s 2024-04-06T02:49:19.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:19 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/2455319487' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-06T02:49:19.962 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-06T02:49:19.962 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-06T02:49:19.962 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi184.front.sepia.ceph.com 2024-04-06T02:49:19.962 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-06T02:49:19.962 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-06T02:49:19.962 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-06T02:49:19.963 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-06T02:49:19.963 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-06T02:49:19.963 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-06T02:49:19.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T02:49:19.963 DEBUG:teuthology.orchestra.run.smithi184:> ip addr 2024-04-06T02:49:19.983 INFO:teuthology.orchestra.run.smithi184.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-06T02:49:19.983 INFO:teuthology.orchestra.run.smithi184.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-06T02:49:19.983 INFO:teuthology.orchestra.run.smithi184.stdout: inet 127.0.0.1/8 scope host lo 2024-04-06T02:49:19.983 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-04-06T02:49:19.983 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 ::1/128 scope host 2024-04-06T02:49:19.983 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-04-06T02:49:19.983 INFO:teuthology.orchestra.run.smithi184.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T02:49:19.983 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:16 brd ff:ff:ff:ff:ff:ff 2024-04-06T02:49:19.984 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f0 2024-04-06T02:49:19.984 INFO:teuthology.orchestra.run.smithi184.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T02:49:19.984 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:17 brd ff:ff:ff:ff:ff:ff 2024-04-06T02:49:19.984 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f1 2024-04-06T02:49:19.984 INFO:teuthology.orchestra.run.smithi184.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-06T02:49:19.984 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8e brd ff:ff:ff:ff:ff:ff 2024-04-06T02:49:19.984 INFO:teuthology.orchestra.run.smithi184.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-06T02:49:19.984 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8f brd ff:ff:ff:ff:ff:ff 2024-04-06T02:49:19.984 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.21.15.184/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-06T02:49:19.984 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft 42256sec preferred_lft 42256sec 2024-04-06T02:49:19.984 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::ec4:7aff:fe88:7b8f/64 scope link 2024-04-06T02:49:19.984 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-04-06T02:49:19.984 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-06T02:49:19.985 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-06T02:49:19.985 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-04-06T02:49:19.985 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link add name ceph-brx type bridge 2024-04-06T02:49:19.985 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip addr flush dev ceph-brx 2024-04-06T02:49:19.985 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link set ceph-brx up 2024-04-06T02:49:19.985 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-06T02:49:19.985 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-04-06T02:49:20.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:49:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-06T02:49:20.089 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:19 smithi184 ceph-mon[37439]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.0 KiB/s rd, 85 B/s wr, 3 op/s 2024-04-06T02:49:20.089 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:19 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/2455319487' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-06T02:49:20.187 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:49:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-06T02:49:20.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T02:49:20.194 DEBUG:teuthology.orchestra.run.smithi184:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-06T02:49:20.222 INFO:teuthology.orchestra.run.smithi184.stdout:1 2024-04-06T02:49:20.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T02:49:20.224 DEBUG:teuthology.orchestra.run.smithi184:> route 2024-04-06T02:49:20.292 INFO:teuthology.orchestra.run.smithi184.stdout:Kernel IP routing table 2024-04-06T02:49:20.292 INFO:teuthology.orchestra.run.smithi184.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-06T02:49:20.292 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-06T02:49:20.292 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-06T02:49:20.292 INFO:teuthology.orchestra.run.smithi184.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-06T02:49:20.292 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-06T02:49:20.292 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-04-06T02:49:20.292 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-06T02:49:20.293 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-06T02:49:20.293 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-06T02:49:20.293 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-04-06T02:49:20.380 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:49:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-06T02:49:20.580 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:49:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-06T02:49:20.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T02:49:20.586 DEBUG:teuthology.orchestra.run.smithi184:> ip netns list 2024-04-06T02:49:20.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T02:49:20.600 DEBUG:teuthology.orchestra.run.smithi184:> ip netns list-id 2024-04-06T02:49:20.655 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-06T02:49:20.655 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-04-06T02:49:20.655 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-06T02:49:20.655 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-06T02:49:20.655 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-04-06T02:49:20.743 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:49:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-06T02:49:20.771 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:49:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-06T02:49:20.777 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-06T02:49:20.778 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-06T02:49:20.778 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-04-06T02:49:20.778 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-06T02:49:20.778 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-06T02:49:20.778 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-06T02:49:20.778 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-06T02:49:20.778 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-06T02:49:20.778 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-04-06T02:49:20.826 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:49:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-06T02:49:20.909 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:49:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-06T02:49:20.915 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-06T02:49:20.915 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-04-06T02:49:20.915 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link set brx.0 up 2024-04-06T02:49:20.915 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link set dev brx.0 master ceph-brx 2024-04-06T02:49:20.915 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-04-06T02:49:20.962 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:49:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-06T02:49:20.990 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:49:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-06T02:49:20.996 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T02:49:20.996 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-06T02:49:21.011 INFO:teuthology.orchestra.run.smithi184.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-06T02:49:21.011 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-06T02:49:21.011 DEBUG:teuthology.orchestra.run.smithi184:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-06T02:49:21.065 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-06T02:49:21.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T02:49:21.066 DEBUG:teuthology.orchestra.run.smithi184:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-06T02:49:21.160 INFO:teuthology.orchestra.run.smithi184.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-06T02:49:21.161 INFO:teuthology.orchestra.run.smithi184.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-04-06T02:49:21.161 INFO:teuthology.orchestra.run.smithi184.stdout:invalid new device string format 2024-04-06T02:49:21.272 INFO:teuthology.orchestra.run.smithi184.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-06T02:49:21.272 INFO:teuthology.orchestra.run.smithi184.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-04-06T02:49:21.272 INFO:teuthology.orchestra.run.smithi184.stdout:invalid new device string format 2024-04-06T02:49:21.273 INFO:teuthology.orchestra.run.smithi184.stdout:mount.ceph: resolved to: "172.21.15.105:3300,172.21.15.184:3300" 2024-04-06T02:49:21.273 INFO:teuthology.orchestra.run.smithi184.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T02:49:21.277 INFO:teuthology.orchestra.run.smithi184.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-06T02:49:21.277 INFO:teuthology.orchestra.run.smithi184.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-04-06T02:49:21.277 INFO:teuthology.orchestra.run.smithi184.stdout:invalid new device string format 2024-04-06T02:49:21.277 INFO:teuthology.orchestra.run.smithi184.stdout:mount.ceph: resolved to: "172.21.15.105:3300,172.21.15.184:3300" 2024-04-06T02:49:21.277 INFO:teuthology.orchestra.run.smithi184.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-06T02:49:21.277 INFO:teuthology.orchestra.run.smithi184.stdout:mount.ceph: resolved to: "172.21.15.105:6789,172.21.15.184:6789" 2024-04-06T02:49:21.277 INFO:teuthology.orchestra.run.smithi184.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.105:6789,172.21.15.184:6789:/ 2024-04-06T02:49:21.277 INFO:teuthology.orchestra.run.smithi184.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=130df578-f3bf-11ee-b649-cb9ed24678a4" will pass to kernel 2024-04-06T02:49:21.278 INFO:teuthology.orchestra.run.smithi184.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-06T02:49:21.278 INFO:teuthology.orchestra.run.smithi184.stdout: You just mounted a file system that supports labels which does not 2024-04-06T02:49:21.279 INFO:teuthology.orchestra.run.smithi184.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-06T02:49:21.279 INFO:teuthology.orchestra.run.smithi184.stdout: applications will generate AVC messages and not be allowed access to 2024-04-06T02:49:21.279 INFO:teuthology.orchestra.run.smithi184.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-06T02:49:21.280 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-06T02:49:21.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T02:49:21.281 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-06T02:49:21.307 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T02:49:21.404 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:49:21 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-06T02:49:21.406 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:49:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-06T02:49:21.411 INFO:teuthology.orchestra.run.smithi184.stdout:client24301 2024-04-06T02:49:21.411 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-04-06T02:49:21.412 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T02:49:21.428 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T02:49:21.517 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:49:21 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-06T02:49:21.518 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:49:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-06T02:49:21.524 INFO:teuthology.orchestra.run.smithi184.stdout:client24301 2024-04-06T02:49:21.524 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-06T02:49:21.572 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:49:21 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-06T02:49:21.573 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:49:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-06T02:49:21.579 INFO:teuthology.orchestra.run.smithi184.stdout:client24301 2024-04-06T02:49:21.579 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4.client24301/status') 2024-04-06T02:49:21.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:49:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-06T02:49:21.640 INFO:teuthology.orchestra.run.smithi184.stdout:instance: client.24301 (3)192.168.0.1:0/1268587846 2024-04-06T02:49:21.640 INFO:teuthology.orchestra.run.smithi184.stdout:blocklisted: false 2024-04-06T02:49:21.640 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2024-04-06T02:49:21.641 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2024-04-06T02:49:21.641 INFO:teuthology.orchestra.run.smithi184.stderr:70 bytes copied, 2.1799e-05 s, 3.2 MB/s 2024-04-06T02:49:21.641 INFO:teuthology.orchestra.run.smithi184.stderr:2024-04-06T02:49:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-06T02:49:21.648 INFO:teuthology.run_tasks:Running task print... 2024-04-06T02:49:21.659 INFO:teuthology.task.print:**** done client 2024-04-06T02:49:21.659 INFO:teuthology.run_tasks:Running task parallel... 2024-04-06T02:49:21.719 INFO:teuthology.task.parallel:starting parallel... 2024-04-06T02:49:21.719 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-06T02:49:21.720 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-06T02:49:21.720 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi105.front.sepia.ceph.com 2024-04-06T02:49:21.720 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-06T02:49:21.721 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-06T02:49:21.721 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-04-06T02:49:21.724 INFO:tasks.workunit:Pulling workunits from ref 584a67ccb9eb224c5312dccd96a82e67b0645518 2024-04-06T02:49:21.725 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-06T02:49:21.725 INFO:tasks.workunit:timeout=3h 2024-04-06T02:49:21.725 INFO:tasks.workunit:cleanup=True 2024-04-06T02:49:21.725 DEBUG:teuthology.orchestra.run.smithi105:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-06T02:49:21.744 INFO:teuthology.orchestra.run.smithi105.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-06T02:49:21.744 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-06T02:49:21.745 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 94h/148d Inode: 1 Links: 2 2024-04-06T02:49:21.745 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T02:49:21.745 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-06T02:49:21.745 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-06T02:49:21.745 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-04-06 02:48:47.490083602 +0000 2024-04-06T02:49:21.745 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-04-06 02:49:16.740948252 +0000 2024-04-06T02:49:21.745 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-04-06 02:48:47.490083602 +0000 2024-04-06T02:49:21.745 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-06T02:49:21.745 DEBUG:teuthology.orchestra.run.smithi105:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-06T02:49:21.819 DEBUG:teuthology.orchestra.run.smithi184:> stat -- /home/ubuntu/cephtest/mnt.1 2024-04-06T02:49:21.836 INFO:teuthology.orchestra.run.smithi184.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-04-06T02:49:21.836 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-04-06T02:49:21.837 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 7eh/126d Inode: 1 Links: 3 2024-04-06T02:49:21.837 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T02:49:21.837 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-06T02:49:21.837 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-06T02:49:21.837 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-04-06 02:49:21.827846850 +0000 2024-04-06T02:49:21.837 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-04-06 02:49:21.827846850 +0000 2024-04-06T02:49:21.837 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-04-06 02:48:47.490083602 +0000 2024-04-06T02:49:21.837 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-04-06T02:49:21.837 DEBUG:teuthology.orchestra.run.smithi184:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-04-06T02:49:21.900 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:21 smithi184 ceph-mon[37439]: pgmap v198: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-06T02:49:21.910 DEBUG:teuthology.orchestra.run.smithi105:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 584a67ccb9eb224c5312dccd96a82e67b0645518 2024-04-06T02:49:21.910 DEBUG:teuthology.orchestra.run.smithi184:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 584a67ccb9eb224c5312dccd96a82e67b0645518 2024-04-06T02:49:21.925 INFO:tasks.workunit.client.0.smithi105.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-06T02:49:21.951 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:49:21.965 INFO:tasks.workunit.client.1.smithi184.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-04-06T02:49:22.073 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:21 smithi105 ceph-mon[29228]: pgmap v198: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-06T02:49:23.616 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:23 smithi105 ceph-mon[29228]: pgmap v199: 65 pgs: 65 active+clean; 582 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-06T02:49:23.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.654+0000 7f6c989e8700 1 -- 172.21.15.105:0/2346871112 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c94101db0 msgr2=0x7f6c94102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:23.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.654+0000 7f6c989e8700 1 --2- 172.21.15.105:0/2346871112 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c94101db0 0x7f6c94102210 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6c88009910 tx=0x7f6c88009c20 comp rx=0 tx=0).stop 2024-04-06T02:49:23.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.654+0000 7f6c989e8700 1 -- 172.21.15.105:0/2346871112 shutdown_connections 2024-04-06T02:49:23.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.654+0000 7f6c989e8700 1 --2- 172.21.15.105:0/2346871112 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c94101db0 0x7f6c94102210 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:23.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.654+0000 7f6c989e8700 1 --2- 172.21.15.105:0/2346871112 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c94100bb0 0x7f6c94100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:23.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.654+0000 7f6c989e8700 1 -- 172.21.15.105:0/2346871112 >> 172.21.15.105:0/2346871112 conn(0x7f6c940fc110 msgr2=0x7f6c940fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:49:23.641 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.655+0000 7f6c989e8700 1 -- 172.21.15.105:0/2346871112 shutdown_connections 2024-04-06T02:49:23.641 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.655+0000 7f6c989e8700 1 -- 172.21.15.105:0/2346871112 wait complete. 2024-04-06T02:49:23.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.655+0000 7f6c989e8700 1 Processor -- start 2024-04-06T02:49:23.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.655+0000 7f6c989e8700 1 -- start start 2024-04-06T02:49:23.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.655+0000 7f6c989e8700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c94100bb0 0x7f6c94196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:23.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.655+0000 7f6c989e8700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c94101db0 0x7f6c94196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:23.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.655+0000 7f6c989e8700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c94196f60 con 0x7f6c94100bb0 2024-04-06T02:49:23.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.655+0000 7f6c989e8700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c941970a0 con 0x7f6c94101db0 2024-04-06T02:49:23.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.656+0000 7f6c9259c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c94100bb0 0x7f6c94196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:23.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.656+0000 7f6c9259c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c94100bb0 0x7f6c94196400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:44406/0 (socket says 172.21.15.105:44406) 2024-04-06T02:49:23.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.656+0000 7f6c9259c700 1 -- 172.21.15.105:0/3663740593 learned_addr learned my addr 172.21.15.105:0/3663740593 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:49:23.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.656+0000 7f6c91d9b700 1 --2- 172.21.15.105:0/3663740593 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c94101db0 0x7f6c94196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:23.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.656+0000 7f6c9259c700 1 -- 172.21.15.105:0/3663740593 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c94101db0 msgr2=0x7f6c94196940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:23.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.656+0000 7f6c9259c700 1 --2- 172.21.15.105:0/3663740593 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c94101db0 0x7f6c94196940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:23.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.656+0000 7f6c9259c700 1 -- 172.21.15.105:0/3663740593 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c880095f0 con 0x7f6c94100bb0 2024-04-06T02:49:23.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.656+0000 7f6c9259c700 1 --2- 172.21.15.105:0/3663740593 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c94100bb0 0x7f6c94196400 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6c8400b480 tx=0x7f6c8400b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:23.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.656+0000 7f6c837fe700 1 -- 172.21.15.105:0/3663740593 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c84011800 con 0x7f6c94100bb0 2024-04-06T02:49:23.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.656+0000 7f6c989e8700 1 -- 172.21.15.105:0/3663740593 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c9419bb50 con 0x7f6c94100bb0 2024-04-06T02:49:23.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.656+0000 7f6c837fe700 1 -- 172.21.15.105:0/3663740593 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6c84011e10 con 0x7f6c94100bb0 2024-04-06T02:49:23.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.656+0000 7f6c837fe700 1 -- 172.21.15.105:0/3663740593 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c84010ea0 con 0x7f6c94100bb0 2024-04-06T02:49:23.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.656+0000 7f6c989e8700 1 -- 172.21.15.105:0/3663740593 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c9419c0a0 con 0x7f6c94100bb0 2024-04-06T02:49:23.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.657+0000 7f6c989e8700 1 -- 172.21.15.105:0/3663740593 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c941097b0 con 0x7f6c94100bb0 2024-04-06T02:49:23.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.658+0000 7f6c837fe700 1 -- 172.21.15.105:0/3663740593 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f6c84010380 con 0x7f6c94100bb0 2024-04-06T02:49:23.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.658+0000 7f6c837fe700 1 --2- 172.21.15.105:0/3663740593 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f6c7c06c4e0 0x7f6c7c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:23.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.658+0000 7f6c837fe700 1 -- 172.21.15.105:0/3663740593 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f6c8408b400 con 0x7f6c94100bb0 2024-04-06T02:49:23.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.658+0000 7f6c91d9b700 1 --2- 172.21.15.105:0/3663740593 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f6c7c06c4e0 0x7f6c7c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:23.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.659+0000 7f6c91d9b700 1 --2- 172.21.15.105:0/3663740593 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f6c7c06c4e0 0x7f6c7c06e9a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6c88006010 tx=0x7f6c8800b560 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:23.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.663+0000 7f6c837fe700 1 -- 172.21.15.105:0/3663740593 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6c840596c0 con 0x7f6c94100bb0 2024-04-06T02:49:23.845 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.859+0000 7f6c989e8700 1 -- 172.21.15.105:0/3663740593 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f6c941099a0 con 0x7f6c94100bb0 2024-04-06T02:49:23.845 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.859+0000 7f6c837fe700 1 -- 172.21.15.105:0/3663740593 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7f6c84059250 con 0x7f6c94100bb0 2024-04-06T02:49:23.847 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.861+0000 7f6c989e8700 1 -- 172.21.15.105:0/3663740593 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f6c7c06c4e0 msgr2=0x7f6c7c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:23.847 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.861+0000 7f6c989e8700 1 --2- 172.21.15.105:0/3663740593 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f6c7c06c4e0 0x7f6c7c06e9a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6c88006010 tx=0x7f6c8800b560 comp rx=0 tx=0).stop 2024-04-06T02:49:23.847 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.861+0000 7f6c989e8700 1 -- 172.21.15.105:0/3663740593 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c94100bb0 msgr2=0x7f6c94196400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:23.847 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.861+0000 7f6c989e8700 1 --2- 172.21.15.105:0/3663740593 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c94100bb0 0x7f6c94196400 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6c8400b480 tx=0x7f6c8400b790 comp rx=0 tx=0).stop 2024-04-06T02:49:23.847 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.861+0000 7f6c989e8700 1 -- 172.21.15.105:0/3663740593 shutdown_connections 2024-04-06T02:49:23.847 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.861+0000 7f6c989e8700 1 --2- 172.21.15.105:0/3663740593 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f6c7c06c4e0 0x7f6c7c06e9a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:23.847 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.861+0000 7f6c989e8700 1 --2- 172.21.15.105:0/3663740593 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c94100bb0 0x7f6c94196400 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:23.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.861+0000 7f6c989e8700 1 --2- 172.21.15.105:0/3663740593 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c94101db0 0x7f6c94196940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:23.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.861+0000 7f6c989e8700 1 -- 172.21.15.105:0/3663740593 >> 172.21.15.105:0/3663740593 conn(0x7f6c940fc110 msgr2=0x7f6c94104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:49:23.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.861+0000 7f6c989e8700 1 -- 172.21.15.105:0/3663740593 shutdown_connections 2024-04-06T02:49:23.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:23.861+0000 7f6c989e8700 1 -- 172.21.15.105:0/3663740593 wait complete. 2024-04-06T02:49:23.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:23 smithi184 ceph-mon[37439]: pgmap v199: 65 pgs: 65 active+clean; 582 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-06T02:49:24.532 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-06T02:49:24.783 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:49:24.904 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:24 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:49:24.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:24 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:49:25.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:25 smithi105 ceph-mon[29228]: pgmap v200: 65 pgs: 65 active+clean; 582 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-06T02:49:25.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:25 smithi184 ceph-mon[37439]: pgmap v200: 65 pgs: 65 active+clean; 582 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-06T02:49:26.638 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.616+0000 7f799ba2b700 1 -- 172.21.15.105:0/989191099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7994100ce0 msgr2=0x7f7994101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:26.639 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.616+0000 7f799ba2b700 1 --2- 172.21.15.105:0/989191099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7994100ce0 0x7f7994101140 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7984009910 tx=0x7f7984009c20 comp rx=0 tx=0).stop 2024-04-06T02:49:26.639 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.616+0000 7f799ba2b700 1 -- 172.21.15.105:0/989191099 shutdown_connections 2024-04-06T02:49:26.639 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.616+0000 7f799ba2b700 1 --2- 172.21.15.105:0/989191099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7994100ce0 0x7f7994101140 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:26.639 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.616+0000 7f799ba2b700 1 --2- 172.21.15.105:0/989191099 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79940ffae0 0x7f79940fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:26.639 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.616+0000 7f799ba2b700 1 -- 172.21.15.105:0/989191099 >> 172.21.15.105:0/989191099 conn(0x7f79940fb080 msgr2=0x7f79940fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:49:26.639 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.616+0000 7f799ba2b700 1 -- 172.21.15.105:0/989191099 shutdown_connections 2024-04-06T02:49:26.639 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.616+0000 7f799ba2b700 1 -- 172.21.15.105:0/989191099 wait complete. 2024-04-06T02:49:26.639 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.617+0000 7f799ba2b700 1 Processor -- start 2024-04-06T02:49:26.639 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.617+0000 7f799ba2b700 1 -- start start 2024-04-06T02:49:26.639 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.617+0000 7f799ba2b700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79940ffae0 0x7f7994103a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:26.639 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.617+0000 7f799ba2b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7994100ce0 0x7f79941020c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:26.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.617+0000 7f799ba2b700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7994102600 con 0x7f7994100ce0 2024-04-06T02:49:26.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.617+0000 7f799ba2b700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7994102740 con 0x7f79940ffae0 2024-04-06T02:49:26.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.617+0000 7f7998fc6700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7994100ce0 0x7f79941020c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:26.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.617+0000 7f7998fc6700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7994100ce0 0x7f79941020c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:44426/0 (socket says 172.21.15.105:44426) 2024-04-06T02:49:26.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.617+0000 7f7998fc6700 1 -- 172.21.15.105:0/3208081804 learned_addr learned my addr 172.21.15.105:0/3208081804 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:49:26.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.618+0000 7f7998fc6700 1 -- 172.21.15.105:0/3208081804 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79940ffae0 msgr2=0x7f7994103a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:26.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.618+0000 7f7998fc6700 1 --2- 172.21.15.105:0/3208081804 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79940ffae0 0x7f7994103a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:26.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.618+0000 7f7998fc6700 1 -- 172.21.15.105:0/3208081804 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79840095f0 con 0x7f7994100ce0 2024-04-06T02:49:26.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.618+0000 7f7998fc6700 1 --2- 172.21.15.105:0/3208081804 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7994100ce0 0x7f79941020c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7984006010 tx=0x7f798400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:26.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.618+0000 7f798a7fc700 1 -- 172.21.15.105:0/3208081804 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f798401d070 con 0x7f7994100ce0 2024-04-06T02:49:26.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.618+0000 7f798a7fc700 1 -- 172.21.15.105:0/3208081804 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f798400f460 con 0x7f7994100ce0 2024-04-06T02:49:26.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.618+0000 7f799ba2b700 1 -- 172.21.15.105:0/3208081804 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79941029c0 con 0x7f7994100ce0 2024-04-06T02:49:26.641 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.618+0000 7f798a7fc700 1 -- 172.21.15.105:0/3208081804 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7984021640 con 0x7f7994100ce0 2024-04-06T02:49:26.641 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.618+0000 7f799ba2b700 1 -- 172.21.15.105:0/3208081804 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7994102f30 con 0x7f7994100ce0 2024-04-06T02:49:26.641 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.619+0000 7f799ba2b700 1 -- 172.21.15.105:0/3208081804 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79941086e0 con 0x7f7994100ce0 2024-04-06T02:49:26.641 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.619+0000 7f798a7fc700 1 -- 172.21.15.105:0/3208081804 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f798400fab0 con 0x7f7994100ce0 2024-04-06T02:49:26.641 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.620+0000 7f798a7fc700 1 --2- 172.21.15.105:0/3208081804 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f798006c4e0 0x7f798006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:26.641 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.620+0000 7f798a7fc700 1 -- 172.21.15.105:0/3208081804 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f798408cba0 con 0x7f7994100ce0 2024-04-06T02:49:26.641 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.620+0000 7f79997c7700 1 --2- 172.21.15.105:0/3208081804 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f798006c4e0 0x7f798006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:26.641 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.620+0000 7f79997c7700 1 --2- 172.21.15.105:0/3208081804 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f798006c4e0 0x7f798006e9a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7994100b40 tx=0x7f7990006d20 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:26.641 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.624+0000 7f798a7fc700 1 -- 172.21.15.105:0/3208081804 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f798405adb0 con 0x7f7994100ce0 2024-04-06T02:49:26.799 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.813+0000 7f799ba2b700 1 -- 172.21.15.105:0/3208081804 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f7994102f30 con 0x7f7994100ce0 2024-04-06T02:49:26.799 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.814+0000 7f798a7fc700 1 -- 172.21.15.105:0/3208081804 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f7994102f30 con 0x7f7994100ce0 2024-04-06T02:49:26.801 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.816+0000 7f799ba2b700 1 -- 172.21.15.105:0/3208081804 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f798006c4e0 msgr2=0x7f798006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:26.802 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.816+0000 7f799ba2b700 1 --2- 172.21.15.105:0/3208081804 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f798006c4e0 0x7f798006e9a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7994100b40 tx=0x7f7990006d20 comp rx=0 tx=0).stop 2024-04-06T02:49:26.802 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.816+0000 7f799ba2b700 1 -- 172.21.15.105:0/3208081804 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7994100ce0 msgr2=0x7f79941020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:26.802 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.816+0000 7f799ba2b700 1 --2- 172.21.15.105:0/3208081804 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7994100ce0 0x7f79941020c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7984006010 tx=0x7f798400be20 comp rx=0 tx=0).stop 2024-04-06T02:49:26.802 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.816+0000 7f799ba2b700 1 -- 172.21.15.105:0/3208081804 shutdown_connections 2024-04-06T02:49:26.802 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.816+0000 7f799ba2b700 1 --2- 172.21.15.105:0/3208081804 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f798006c4e0 0x7f798006e9a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:26.802 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.816+0000 7f799ba2b700 1 --2- 172.21.15.105:0/3208081804 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79940ffae0 0x7f7994103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:26.802 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.816+0000 7f799ba2b700 1 --2- 172.21.15.105:0/3208081804 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7994100ce0 0x7f79941020c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:26.802 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.816+0000 7f799ba2b700 1 -- 172.21.15.105:0/3208081804 >> 172.21.15.105:0/3208081804 conn(0x7f79940fb080 msgr2=0x7f7994068dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:49:26.802 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.816+0000 7f799ba2b700 1 -- 172.21.15.105:0/3208081804 shutdown_connections 2024-04-06T02:49:26.802 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:26.816+0000 7f799ba2b700 1 -- 172.21.15.105:0/3208081804 wait complete. 2024-04-06T02:49:27.457 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-06T02:49:27.665 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:49:27.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:27 smithi184 ceph-mon[37439]: pgmap v201: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s wr, 0 op/s 2024-04-06T02:49:28.004 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:27 smithi105 ceph-mon[29228]: pgmap v201: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s wr, 0 op/s 2024-04-06T02:49:29.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.372+0000 7fc4b3a09700 1 -- 172.21.15.105:0/4080480063 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4ac100cb0 msgr2=0x7fc4ac101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:29.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.372+0000 7fc4b3a09700 1 --2- 172.21.15.105:0/4080480063 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4ac100cb0 0x7fc4ac101110 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc49c009910 tx=0x7fc49c009c20 comp rx=0 tx=0).stop 2024-04-06T02:49:29.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.372+0000 7fc4b3a09700 1 -- 172.21.15.105:0/4080480063 shutdown_connections 2024-04-06T02:49:29.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.372+0000 7fc4b3a09700 1 --2- 172.21.15.105:0/4080480063 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4ac100cb0 0x7fc4ac101110 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:29.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.372+0000 7fc4b3a09700 1 --2- 172.21.15.105:0/4080480063 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4ac0ffab0 0x7fc4ac0ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:29.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.372+0000 7fc4b3a09700 1 -- 172.21.15.105:0/4080480063 >> 172.21.15.105:0/4080480063 conn(0x7fc4ac0fb030 msgr2=0x7fc4ac0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:49:29.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.372+0000 7fc4b3a09700 1 -- 172.21.15.105:0/4080480063 shutdown_connections 2024-04-06T02:49:29.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.372+0000 7fc4b3a09700 1 -- 172.21.15.105:0/4080480063 wait complete. 2024-04-06T02:49:29.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.373+0000 7fc4b3a09700 1 Processor -- start 2024-04-06T02:49:29.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.373+0000 7fc4b3a09700 1 -- start start 2024-04-06T02:49:29.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.373+0000 7fc4b3a09700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4ac0ffab0 0x7fc4ac068f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:29.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.373+0000 7fc4b3a09700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4ac100cb0 0x7fc4ac19b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:29.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.373+0000 7fc4b3a09700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac19b710 con 0x7fc4ac0ffab0 2024-04-06T02:49:29.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.373+0000 7fc4b3a09700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac19b850 con 0x7fc4ac100cb0 2024-04-06T02:49:29.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.373+0000 7fc4b17a5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4ac0ffab0 0x7fc4ac068f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:29.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.373+0000 7fc4b17a5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4ac0ffab0 0x7fc4ac068f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:44432/0 (socket says 172.21.15.105:44432) 2024-04-06T02:49:29.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.373+0000 7fc4b17a5700 1 -- 172.21.15.105:0/3583467102 learned_addr learned my addr 172.21.15.105:0/3583467102 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:49:29.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.373+0000 7fc4b0fa4700 1 --2- 172.21.15.105:0/3583467102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4ac100cb0 0x7fc4ac19b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:29.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.373+0000 7fc4b17a5700 1 -- 172.21.15.105:0/3583467102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4ac100cb0 msgr2=0x7fc4ac19b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:29.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.373+0000 7fc4b17a5700 1 --2- 172.21.15.105:0/3583467102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4ac100cb0 0x7fc4ac19b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:29.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.374+0000 7fc4b17a5700 1 -- 172.21.15.105:0/3583467102 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc49c0095f0 con 0x7fc4ac0ffab0 2024-04-06T02:49:29.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.374+0000 7fc4b0fa4700 1 --2- 172.21.15.105:0/3583467102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4ac100cb0 0x7fc4ac19b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-06T02:49:29.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.374+0000 7fc4b17a5700 1 --2- 172.21.15.105:0/3583467102 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4ac0ffab0 0x7fc4ac068f70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc4a800b480 tx=0x7fc4a800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:29.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.374+0000 7fc4a27fc700 1 -- 172.21.15.105:0/3583467102 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4a80118c0 con 0x7fc4ac0ffab0 2024-04-06T02:49:29.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.374+0000 7fc4a27fc700 1 -- 172.21.15.105:0/3583467102 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc4a8010380 con 0x7fc4ac0ffab0 2024-04-06T02:49:29.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.374+0000 7fc4b3a09700 1 -- 172.21.15.105:0/3583467102 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4ac19bb30 con 0x7fc4ac0ffab0 2024-04-06T02:49:29.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.374+0000 7fc4a27fc700 1 -- 172.21.15.105:0/3583467102 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4a800f4f0 con 0x7fc4ac0ffab0 2024-04-06T02:49:29.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.374+0000 7fc4b3a09700 1 -- 172.21.15.105:0/3583467102 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4ac19c100 con 0x7fc4ac0ffab0 2024-04-06T02:49:29.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.375+0000 7fc4b3a09700 1 -- 172.21.15.105:0/3583467102 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4ac1086b0 con 0x7fc4ac0ffab0 2024-04-06T02:49:29.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.375+0000 7fc4a27fc700 1 -- 172.21.15.105:0/3583467102 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fc4a80104f0 con 0x7fc4ac0ffab0 2024-04-06T02:49:29.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.376+0000 7fc4a27fc700 1 --2- 172.21.15.105:0/3583467102 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fc49806c4e0 0x7fc49806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:29.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.376+0000 7fc4a27fc700 1 -- 172.21.15.105:0/3583467102 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fc4a808b540 con 0x7fc4ac0ffab0 2024-04-06T02:49:29.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.376+0000 7fc4b0fa4700 1 --2- 172.21.15.105:0/3583467102 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fc49806c4e0 0x7fc49806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:29.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.376+0000 7fc4b0fa4700 1 --2- 172.21.15.105:0/3583467102 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fc49806c4e0 0x7fc49806e9a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc49c00b5c0 tx=0x7fc49c005e20 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:29.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.380+0000 7fc4a27fc700 1 -- 172.21.15.105:0/3583467102 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc4a8059750 con 0x7fc4ac0ffab0 2024-04-06T02:49:29.554 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.568+0000 7fc4b3a09700 1 -- 172.21.15.105:0/3583467102 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7fc4ac19c910 con 0x7fc4ac0ffab0 2024-04-06T02:49:29.554 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.569+0000 7fc4a27fc700 1 -- 172.21.15.105:0/3583467102 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7fc4a80592e0 con 0x7fc4ac0ffab0 2024-04-06T02:49:29.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.571+0000 7fc4b3a09700 1 -- 172.21.15.105:0/3583467102 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fc49806c4e0 msgr2=0x7fc49806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:29.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.571+0000 7fc4b3a09700 1 --2- 172.21.15.105:0/3583467102 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fc49806c4e0 0x7fc49806e9a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc49c00b5c0 tx=0x7fc49c005e20 comp rx=0 tx=0).stop 2024-04-06T02:49:29.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.571+0000 7fc4b3a09700 1 -- 172.21.15.105:0/3583467102 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4ac0ffab0 msgr2=0x7fc4ac068f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:29.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.571+0000 7fc4b3a09700 1 --2- 172.21.15.105:0/3583467102 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4ac0ffab0 0x7fc4ac068f70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc4a800b480 tx=0x7fc4a800b840 comp rx=0 tx=0).stop 2024-04-06T02:49:29.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.571+0000 7fc4b3a09700 1 -- 172.21.15.105:0/3583467102 shutdown_connections 2024-04-06T02:49:29.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.571+0000 7fc4b3a09700 1 --2- 172.21.15.105:0/3583467102 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fc49806c4e0 0x7fc49806e9a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:29.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.571+0000 7fc4b3a09700 1 --2- 172.21.15.105:0/3583467102 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4ac0ffab0 0x7fc4ac068f70 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:29.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.571+0000 7fc4b3a09700 1 --2- 172.21.15.105:0/3583467102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4ac100cb0 0x7fc4ac19b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:29.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.571+0000 7fc4b3a09700 1 -- 172.21.15.105:0/3583467102 >> 172.21.15.105:0/3583467102 conn(0x7fc4ac0fb030 msgr2=0x7fc4ac103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:49:29.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.571+0000 7fc4b3a09700 1 -- 172.21.15.105:0/3583467102 shutdown_connections 2024-04-06T02:49:29.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:29.571+0000 7fc4b3a09700 1 -- 172.21.15.105:0/3583467102 wait complete. 2024-04-06T02:49:29.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:29 smithi184 ceph-mon[37439]: pgmap v202: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-06T02:49:30.006 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:29 smithi105 ceph-mon[29228]: pgmap v202: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-06T02:49:30.422 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-04-06T02:49:30.631 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:49:31.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:31 smithi184 ceph-mon[37439]: pgmap v203: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 0 op/s 2024-04-06T02:49:31.999 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:31 smithi105 ceph-mon[29228]: pgmap v203: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 0 op/s 2024-04-06T02:49:32.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.461+0000 7fe333932700 1 -- 172.21.15.105:0/4159676240 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe32c101dc0 msgr2=0x7fe32c102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:32.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.461+0000 7fe333932700 1 --2- 172.21.15.105:0/4159676240 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe32c101dc0 0x7fe32c102220 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe31c009910 tx=0x7fe31c009c20 comp rx=0 tx=0).stop 2024-04-06T02:49:32.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.461+0000 7fe333932700 1 -- 172.21.15.105:0/4159676240 shutdown_connections 2024-04-06T02:49:32.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.461+0000 7fe333932700 1 --2- 172.21.15.105:0/4159676240 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe32c101dc0 0x7fe32c102220 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:32.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.461+0000 7fe333932700 1 --2- 172.21.15.105:0/4159676240 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe32c100bc0 0x7fe32c100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:32.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.461+0000 7fe333932700 1 -- 172.21.15.105:0/4159676240 >> 172.21.15.105:0/4159676240 conn(0x7fe32c0fc140 msgr2=0x7fe32c0fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:49:32.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.462+0000 7fe333932700 1 -- 172.21.15.105:0/4159676240 shutdown_connections 2024-04-06T02:49:32.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.462+0000 7fe333932700 1 -- 172.21.15.105:0/4159676240 wait complete. 2024-04-06T02:49:32.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.462+0000 7fe333932700 1 Processor -- start 2024-04-06T02:49:32.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.462+0000 7fe333932700 1 -- start start 2024-04-06T02:49:32.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.462+0000 7fe333932700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe32c100bc0 0x7fe32c074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:32.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.462+0000 7fe333932700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe32c101dc0 0x7fe32c073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:32.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.462+0000 7fe333932700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c0736c0 con 0x7fe32c100bc0 2024-04-06T02:49:32.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.462+0000 7fe333932700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c073800 con 0x7fe32c101dc0 2024-04-06T02:49:32.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.462+0000 7fe3316ce700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe32c100bc0 0x7fe32c074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:32.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.462+0000 7fe3316ce700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe32c100bc0 0x7fe32c074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56524/0 (socket says 172.21.15.105:56524) 2024-04-06T02:49:32.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.462+0000 7fe3316ce700 1 -- 172.21.15.105:0/3188928390 learned_addr learned my addr 172.21.15.105:0/3188928390 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:49:32.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.463+0000 7fe3316ce700 1 -- 172.21.15.105:0/3188928390 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe32c101dc0 msgr2=0x7fe32c073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:49:32.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.463+0000 7fe330ecd700 1 --2- 172.21.15.105:0/3188928390 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe32c101dc0 0x7fe32c073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:32.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.463+0000 7fe3316ce700 1 --2- 172.21.15.105:0/3188928390 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe32c101dc0 0x7fe32c073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:49:32.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.463+0000 7fe3316ce700 1 -- 172.21.15.105:0/3188928390 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe31c0095f0 con 0x7fe32c100bc0 2024-04-06T02:49:32.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.463+0000 7fe3316ce700 1 --2- 172.21.15.105:0/3188928390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe32c100bc0 0x7fe32c074b30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe32800c8f0 tx=0x7fe32800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:32.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.463+0000 7fe3227fc700 1 -- 172.21.15.105:0/3188928390 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe328007a70 con 0x7fe32c100bc0 2024-04-06T02:49:32.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.463+0000 7fe3227fc700 1 -- 172.21.15.105:0/3188928390 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe328007bd0 con 0x7fe32c100bc0 2024-04-06T02:49:32.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.463+0000 7fe333932700 1 -- 172.21.15.105:0/3188928390 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe32c073ae0 con 0x7fe32c100bc0 2024-04-06T02:49:32.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.463+0000 7fe3227fc700 1 -- 172.21.15.105:0/3188928390 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe328018630 con 0x7fe32c100bc0 2024-04-06T02:49:32.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.463+0000 7fe333932700 1 -- 172.21.15.105:0/3188928390 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe32c19e3d0 con 0x7fe32c100bc0 2024-04-06T02:49:32.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.464+0000 7fe333932700 1 -- 172.21.15.105:0/3188928390 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe32c109720 con 0x7fe32c100bc0 2024-04-06T02:49:32.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.465+0000 7fe3227fc700 1 -- 172.21.15.105:0/3188928390 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fe32800f450 con 0x7fe32c100bc0 2024-04-06T02:49:32.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.465+0000 7fe3227fc700 1 --2- 172.21.15.105:0/3188928390 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fe31806c4e0 0x7fe31806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:49:32.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.465+0000 7fe3227fc700 1 -- 172.21.15.105:0/3188928390 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fe32808b890 con 0x7fe32c100bc0 2024-04-06T02:49:32.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.465+0000 7fe330ecd700 1 --2- 172.21.15.105:0/3188928390 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fe31806c4e0 0x7fe31806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:49:32.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.466+0000 7fe330ecd700 1 --2- 172.21.15.105:0/3188928390 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fe31806c4e0 0x7fe31806e9a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe31c00b5c0 tx=0x7fe31c005960 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:49:32.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.470+0000 7fe3227fc700 1 -- 172.21.15.105:0/3188928390 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe328059b50 con 0x7fe32c100bc0 2024-04-06T02:49:32.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:49:32.679+0000 7fe333932700 1 -- 172.21.15.105:0/3188928390 --> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fe32c19e980 con 0x7fe31806c4e0 2024-04-06T02:49:33.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:33 smithi105 ceph-mon[29228]: pgmap v204: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-04-06T02:49:33.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:33 smithi105 ceph-mon[29228]: from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:49:33.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:33 smithi184 ceph-mon[37439]: pgmap v204: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-04-06T02:49:33.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:33 smithi184 ceph-mon[37439]: from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:49:35.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:35 smithi184 ceph-mon[37439]: pgmap v205: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-04-06T02:49:36.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:35 smithi105 ceph-mon[29228]: pgmap v205: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-04-06T02:49:37.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:37 smithi184 ceph-mon[37439]: pgmap v206: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-04-06T02:49:38.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:37 smithi105 ceph-mon[29228]: pgmap v206: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-04-06T02:49:39.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:39 smithi184 ceph-mon[37439]: pgmap v207: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-06T02:49:39.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:39 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:49:40.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:39 smithi105 ceph-mon[29228]: pgmap v207: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-06T02:49:40.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:39 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:49:41.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:41 smithi184 ceph-mon[37439]: pgmap v208: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:49:41.999 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:41 smithi105 ceph-mon[29228]: pgmap v208: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:49:43.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:43 smithi105 ceph-mon[29228]: pgmap v209: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:49:43.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:43 smithi184 ceph-mon[37439]: pgmap v209: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:49:45.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:45 smithi184 ceph-mon[37439]: pgmap v210: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:49:46.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:45 smithi105 ceph-mon[29228]: pgmap v210: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:49:47.971 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:47 smithi184 ceph-mon[37439]: pgmap v211: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:49:48.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:47 smithi105 ceph-mon[29228]: pgmap v211: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:49:49.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:49 smithi184 ceph-mon[37439]: pgmap v212: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:49:50.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:49 smithi105 ceph-mon[29228]: pgmap v212: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:49:51.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:51 smithi184 ceph-mon[37439]: pgmap v213: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:49:52.000 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:51 smithi105 ceph-mon[29228]: pgmap v213: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:49:53.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:53 smithi105 ceph-mon[29228]: pgmap v214: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:49:53.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:53 smithi184 ceph-mon[37439]: pgmap v214: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:49:54.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:54 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:49:55.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:54 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:49:55.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:55 smithi184 ceph-mon[37439]: pgmap v215: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:49:56.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:55 smithi105 ceph-mon[29228]: pgmap v215: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:49:57.731 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:57 smithi184 ceph-mon[37439]: pgmap v216: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-06T02:49:58.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:57 smithi105 ceph-mon[29228]: pgmap v216: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-06T02:49:59.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:49:59 smithi184 ceph-mon[37439]: pgmap v217: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-06T02:50:00.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:49:59 smithi105 ceph-mon[29228]: pgmap v217: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-06T02:50:00.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:00 smithi184 ceph-mon[37439]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-06T02:50:00.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:00 smithi184 ceph-mon[37439]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T02:50:00.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:00 smithi184 ceph-mon[37439]: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T02:50:01.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:00 smithi105 ceph-mon[29228]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-06T02:50:01.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:00 smithi105 ceph-mon[29228]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-06T02:50:01.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:00 smithi105 ceph-mon[29228]: fs cephfs has deprecated feature inline_data enabled. 2024-04-06T02:50:01.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:01 smithi105 ceph-mon[29228]: pgmap v218: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-06T02:50:01.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:01 smithi184 ceph-mon[37439]: pgmap v218: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-06T02:50:03.789 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:03 smithi105 ceph-mon[29228]: pgmap v219: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-04-06T02:50:03.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:03 smithi184 ceph-mon[37439]: pgmap v219: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-04-06T02:50:05.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:05 smithi184 ceph-mon[37439]: pgmap v220: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-04-06T02:50:06.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:05 smithi105 ceph-mon[29228]: pgmap v220: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-04-06T02:50:07.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:07 smithi184 ceph-mon[37439]: pgmap v221: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-04-06T02:50:08.003 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:07 smithi105 ceph-mon[29228]: pgmap v221: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-04-06T02:50:08.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:08.046+0000 7fe3227fc700 1 -- 172.21.15.105:0/3188928390 <== mgr.14221 v2:172.21.15.105:6800/2334550011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7fe32c19e980 con 0x7fe31806c4e0 2024-04-06T02:50:08.032 INFO:teuthology.orchestra.run.smithi105.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T02:50:08.034 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:08.048+0000 7fe333932700 1 -- 172.21.15.105:0/3188928390 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fe31806c4e0 msgr2=0x7fe31806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:08.034 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:08.048+0000 7fe333932700 1 --2- 172.21.15.105:0/3188928390 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fe31806c4e0 0x7fe31806e9a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe31c00b5c0 tx=0x7fe31c005960 comp rx=0 tx=0).stop 2024-04-06T02:50:08.034 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:08.048+0000 7fe333932700 1 -- 172.21.15.105:0/3188928390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe32c100bc0 msgr2=0x7fe32c074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:08.034 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:08.048+0000 7fe333932700 1 --2- 172.21.15.105:0/3188928390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe32c100bc0 0x7fe32c074b30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe32800c8f0 tx=0x7fe32800ccb0 comp rx=0 tx=0).stop 2024-04-06T02:50:08.034 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:08.048+0000 7fe333932700 1 -- 172.21.15.105:0/3188928390 shutdown_connections 2024-04-06T02:50:08.034 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:08.048+0000 7fe333932700 1 --2- 172.21.15.105:0/3188928390 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fe31806c4e0 0x7fe31806e9a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:08.035 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:08.048+0000 7fe333932700 1 --2- 172.21.15.105:0/3188928390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe32c100bc0 0x7fe32c074b30 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:08.035 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:08.048+0000 7fe333932700 1 --2- 172.21.15.105:0/3188928390 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe32c101dc0 0x7fe32c073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:08.035 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:08.048+0000 7fe333932700 1 -- 172.21.15.105:0/3188928390 >> 172.21.15.105:0/3188928390 conn(0x7fe32c0fc140 msgr2=0x7fe32c104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:50:08.035 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:08.048+0000 7fe333932700 1 -- 172.21.15.105:0/3188928390 shutdown_connections 2024-04-06T02:50:08.035 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:08.048+0000 7fe333932700 1 -- 172.21.15.105:0/3188928390 wait complete. 2024-04-06T02:50:08.835 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-04-06T02:50:09.088 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:50:09.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:09 smithi105 ceph-mon[29228]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T02:50:09.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:09 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:50:09.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:09 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:50:09.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:09 smithi105 ceph-mon[29228]: pgmap v222: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-04-06T02:50:09.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:09 smithi184 ceph-mon[37439]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T02:50:09.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:09 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:50:09.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:09 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:50:09.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:09 smithi184 ceph-mon[37439]: pgmap v222: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-04-06T02:50:10.346 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:10 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:50:10.346 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:10 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:50:10.346 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:10 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:50:10.346 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:10 smithi105 ceph-mon[29228]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T02:50:10.346 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:10 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:50:10.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:10 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:50:10.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:10 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T02:50:10.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:10 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:50:10.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:10 smithi184 ceph-mon[37439]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T02:50:10.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:10 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:50:11.423 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:11 smithi105 ceph-mon[29228]: pgmap v223: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-06T02:50:11.481 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:11 smithi184 ceph-mon[37439]: pgmap v223: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-06T02:50:13.676 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:13 smithi105 ceph-mon[29228]: pgmap v224: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-06T02:50:13.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:13 smithi184 ceph-mon[37439]: pgmap v224: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-06T02:50:14.123 INFO:teuthology.orchestra.run.smithi105.stdout:true 2024-04-06T02:50:14.123 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.452+0000 7f50fe8ec700 1 -- 172.21.15.105:0/1830494556 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50f81010a0 msgr2=0x7f50f81014c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:14.123 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.452+0000 7f50fe8ec700 1 --2- 172.21.15.105:0/1830494556 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50f81010a0 0x7f50f81014c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f50e8009910 tx=0x7f50e8009c20 comp rx=0 tx=0).stop 2024-04-06T02:50:14.123 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.452+0000 7f50fe8ec700 1 -- 172.21.15.105:0/1830494556 shutdown_connections 2024-04-06T02:50:14.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.452+0000 7f50fe8ec700 1 --2- 172.21.15.105:0/1830494556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50f80ffd50 0x7f50f81001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:14.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.452+0000 7f50fe8ec700 1 --2- 172.21.15.105:0/1830494556 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50f81010a0 0x7f50f81014c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:14.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.452+0000 7f50fe8ec700 1 -- 172.21.15.105:0/1830494556 >> 172.21.15.105:0/1830494556 conn(0x7f50f80fa7c0 msgr2=0x7f50f80fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:50:14.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.452+0000 7f50fe8ec700 1 -- 172.21.15.105:0/1830494556 shutdown_connections 2024-04-06T02:50:14.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.452+0000 7f50fe8ec700 1 -- 172.21.15.105:0/1830494556 wait complete. 2024-04-06T02:50:14.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.453+0000 7f50fe8ec700 1 Processor -- start 2024-04-06T02:50:14.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.453+0000 7f50fe8ec700 1 -- start start 2024-04-06T02:50:14.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.454+0000 7f50fe8ec700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50f80ffd50 0x7f50f8196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:14.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.454+0000 7f50fe8ec700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50f81010a0 0x7f50f81968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:14.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.454+0000 7f50fe8ec700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f8196ef0 con 0x7f50f81010a0 2024-04-06T02:50:14.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.454+0000 7f50fe8ec700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f8197030 con 0x7f50f80ffd50 2024-04-06T02:50:14.125 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.454+0000 7f50f77fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50f81010a0 0x7f50f81968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:14.125 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.454+0000 7f50f77fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50f81010a0 0x7f50f81968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:58368/0 (socket says 172.21.15.105:58368) 2024-04-06T02:50:14.125 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.454+0000 7f50f77fe700 1 -- 172.21.15.105:0/2591113120 learned_addr learned my addr 172.21.15.105:0/2591113120 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:50:14.125 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.454+0000 7f50f7fff700 1 --2- 172.21.15.105:0/2591113120 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50f80ffd50 0x7f50f8196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:14.125 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.454+0000 7f50f77fe700 1 -- 172.21.15.105:0/2591113120 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50f80ffd50 msgr2=0x7f50f8196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:14.125 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.454+0000 7f50f77fe700 1 --2- 172.21.15.105:0/2591113120 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50f80ffd50 0x7f50f8196390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:14.125 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.454+0000 7f50f77fe700 1 -- 172.21.15.105:0/2591113120 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e80095f0 con 0x7f50f81010a0 2024-04-06T02:50:14.125 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.455+0000 7f50f77fe700 1 --2- 172.21.15.105:0/2591113120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50f81010a0 0x7f50f81968d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f50ec007e80 tx=0x7f50ec00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:50:14.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.455+0000 7f50f57fa700 1 -- 172.21.15.105:0/2591113120 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50ec012c60 con 0x7f50f81010a0 2024-04-06T02:50:14.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.455+0000 7f50fe8ec700 1 -- 172.21.15.105:0/2591113120 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50f8106000 con 0x7f50f81010a0 2024-04-06T02:50:14.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.455+0000 7f50f57fa700 1 -- 172.21.15.105:0/2591113120 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f50ec016070 con 0x7f50f81010a0 2024-04-06T02:50:14.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.455+0000 7f50f57fa700 1 -- 172.21.15.105:0/2591113120 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50ec005020 con 0x7f50f81010a0 2024-04-06T02:50:14.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.455+0000 7f50fe8ec700 1 -- 172.21.15.105:0/2591113120 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50f8106550 con 0x7f50f81010a0 2024-04-06T02:50:14.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.456+0000 7f50fe8ec700 1 -- 172.21.15.105:0/2591113120 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50f8109720 con 0x7f50f81010a0 2024-04-06T02:50:14.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.456+0000 7f50f57fa700 1 -- 172.21.15.105:0/2591113120 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f50ec005360 con 0x7f50f81010a0 2024-04-06T02:50:14.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.457+0000 7f50f57fa700 1 --2- 172.21.15.105:0/2591113120 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f50e0074cf0 0x7f50e00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:14.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.457+0000 7f50f57fa700 1 -- 172.21.15.105:0/2591113120 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f50ec018070 con 0x7f50f81010a0 2024-04-06T02:50:14.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.457+0000 7f50f7fff700 1 --2- 172.21.15.105:0/2591113120 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f50e0074cf0 0x7f50e00771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:14.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.458+0000 7f50f7fff700 1 --2- 172.21.15.105:0/2591113120 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f50e0074cf0 0x7f50e00771b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f50e8006010 tx=0x7f50e800b560 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:50:14.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.461+0000 7f50f57fa700 1 -- 172.21.15.105:0/2591113120 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f50ec0591b0 con 0x7f50f81010a0 2024-04-06T02:50:14.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.708+0000 7f50fe8ec700 1 -- 172.21.15.105:0/2591113120 --> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f50f81056b0 con 0x7f50e0074cf0 2024-04-06T02:50:14.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.710+0000 7f50f57fa700 1 -- 172.21.15.105:0/2591113120 <== mgr.14221 v2:172.21.15.105:6800/2334550011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7f50f81056b0 con 0x7f50e0074cf0 2024-04-06T02:50:14.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.713+0000 7f50fe8ec700 1 -- 172.21.15.105:0/2591113120 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f50e0074cf0 msgr2=0x7f50e00771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:14.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.713+0000 7f50fe8ec700 1 --2- 172.21.15.105:0/2591113120 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f50e0074cf0 0x7f50e00771b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f50e8006010 tx=0x7f50e800b560 comp rx=0 tx=0).stop 2024-04-06T02:50:14.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.713+0000 7f50fe8ec700 1 -- 172.21.15.105:0/2591113120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50f81010a0 msgr2=0x7f50f81968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:14.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.713+0000 7f50fe8ec700 1 --2- 172.21.15.105:0/2591113120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50f81010a0 0x7f50f81968d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f50ec007e80 tx=0x7f50ec00dbe0 comp rx=0 tx=0).stop 2024-04-06T02:50:14.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.713+0000 7f50fe8ec700 1 -- 172.21.15.105:0/2591113120 shutdown_connections 2024-04-06T02:50:14.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.713+0000 7f50fe8ec700 1 --2- 172.21.15.105:0/2591113120 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f50e0074cf0 0x7f50e00771b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:14.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.713+0000 7f50fe8ec700 1 --2- 172.21.15.105:0/2591113120 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50f80ffd50 0x7f50f8196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:14.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.713+0000 7f50fe8ec700 1 --2- 172.21.15.105:0/2591113120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50f81010a0 0x7f50f81968d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:14.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.713+0000 7f50fe8ec700 1 -- 172.21.15.105:0/2591113120 >> 172.21.15.105:0/2591113120 conn(0x7f50f80fa7c0 msgr2=0x7f50f8103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:50:14.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.713+0000 7f50fe8ec700 1 -- 172.21.15.105:0/2591113120 shutdown_connections 2024-04-06T02:50:14.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.713+0000 7f50fe8ec700 1 -- 172.21.15.105:0/2591113120 wait complete. 2024-04-06T02:50:14.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.874+0000 7fa068641700 1 -- 172.21.15.105:0/586483837 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa060100cb0 msgr2=0x7fa060101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:14.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.874+0000 7fa068641700 1 --2- 172.21.15.105:0/586483837 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa060100cb0 0x7fa060101110 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa050009910 tx=0x7fa050009c20 comp rx=0 tx=0).stop 2024-04-06T02:50:14.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.874+0000 7fa068641700 1 -- 172.21.15.105:0/586483837 shutdown_connections 2024-04-06T02:50:14.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.874+0000 7fa068641700 1 --2- 172.21.15.105:0/586483837 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa060100cb0 0x7fa060101110 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:14.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.874+0000 7fa068641700 1 --2- 172.21.15.105:0/586483837 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa0600ffab0 0x7fa0600ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:14.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.874+0000 7fa068641700 1 -- 172.21.15.105:0/586483837 >> 172.21.15.105:0/586483837 conn(0x7fa0600fb030 msgr2=0x7fa0600fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:50:14.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.875+0000 7fa068641700 1 -- 172.21.15.105:0/586483837 shutdown_connections 2024-04-06T02:50:14.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.875+0000 7fa068641700 1 -- 172.21.15.105:0/586483837 wait complete. 2024-04-06T02:50:14.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.875+0000 7fa068641700 1 Processor -- start 2024-04-06T02:50:14.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.875+0000 7fa068641700 1 -- start start 2024-04-06T02:50:14.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.876+0000 7fa068641700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa0600ffab0 0x7fa0601954b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:14.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.876+0000 7fa068641700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa0601959f0 0x7fa06019aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:14.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.876+0000 7fa068641700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa060195f00 con 0x7fa0601959f0 2024-04-06T02:50:14.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.876+0000 7fa068641700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa060196040 con 0x7fa0600ffab0 2024-04-06T02:50:14.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.876+0000 7fa0663dd700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa0600ffab0 0x7fa0601954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:14.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.876+0000 7fa065bdc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa0601959f0 0x7fa06019aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:14.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.876+0000 7fa0663dd700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa0600ffab0 0x7fa0601954b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.105:51468/0 (socket says 172.21.15.105:51468) 2024-04-06T02:50:14.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.876+0000 7fa0663dd700 1 -- 172.21.15.105:0/2505335246 learned_addr learned my addr 172.21.15.105:0/2505335246 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:50:14.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.876+0000 7fa065bdc700 1 -- 172.21.15.105:0/2505335246 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa0600ffab0 msgr2=0x7fa0601954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:14.131 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.876+0000 7fa065bdc700 1 --2- 172.21.15.105:0/2505335246 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa0600ffab0 0x7fa0601954b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:14.131 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.876+0000 7fa065bdc700 1 -- 172.21.15.105:0/2505335246 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0500095f0 con 0x7fa0601959f0 2024-04-06T02:50:14.131 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.876+0000 7fa0663dd700 1 --2- 172.21.15.105:0/2505335246 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa0600ffab0 0x7fa0601954b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T02:50:14.131 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.876+0000 7fa065bdc700 1 --2- 172.21.15.105:0/2505335246 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa0601959f0 0x7fa06019aa60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa050008000 tx=0x7fa0500048c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:50:14.131 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.877+0000 7fa0577fe700 1 -- 172.21.15.105:0/2505335246 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa05001d070 con 0x7fa0601959f0 2024-04-06T02:50:14.131 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.877+0000 7fa0577fe700 1 -- 172.21.15.105:0/2505335246 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa050005dc0 con 0x7fa0601959f0 2024-04-06T02:50:14.131 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.877+0000 7fa068641700 1 -- 172.21.15.105:0/2505335246 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa060068410 con 0x7fa0601959f0 2024-04-06T02:50:14.131 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.877+0000 7fa0577fe700 1 -- 172.21.15.105:0/2505335246 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa05000f460 con 0x7fa0601959f0 2024-04-06T02:50:14.132 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.877+0000 7fa068641700 1 -- 172.21.15.105:0/2505335246 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0600688d0 con 0x7fa0601959f0 2024-04-06T02:50:14.132 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.878+0000 7fa068641700 1 -- 172.21.15.105:0/2505335246 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa060108610 con 0x7fa0601959f0 2024-04-06T02:50:14.132 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.879+0000 7fa0577fe700 1 -- 172.21.15.105:0/2505335246 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fa05000f5c0 con 0x7fa0601959f0 2024-04-06T02:50:14.132 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.880+0000 7fa0577fe700 1 --2- 172.21.15.105:0/2505335246 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fa04c06c530 0x7fa04c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:14.132 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.880+0000 7fa0577fe700 1 -- 172.21.15.105:0/2505335246 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fa05008ced0 con 0x7fa0601959f0 2024-04-06T02:50:14.132 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.880+0000 7fa0663dd700 1 --2- 172.21.15.105:0/2505335246 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fa04c06c530 0x7fa04c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:14.132 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.880+0000 7fa0663dd700 1 --2- 172.21.15.105:0/2505335246 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fa04c06c530 0x7fa04c06e9f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa05c007fd0 tx=0x7fa05c007480 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:50:14.132 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:13.887+0000 7fa0577fe700 1 -- 172.21.15.105:0/2505335246 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa05005c020 con 0x7fa0601959f0 2024-04-06T02:50:14.298 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.312+0000 7fa068641700 1 -- 172.21.15.105:0/2505335246 --> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa060069320 con 0x7fa04c06c530 2024-04-06T02:50:14.299 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.314+0000 7fa0577fe700 1 -- 172.21.15.105:0/2505335246 <== mgr.14221 v2:172.21.15.105:6800/2334550011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7fa060069320 con 0x7fa04c06c530 2024-04-06T02:50:14.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.316+0000 7fa068641700 1 -- 172.21.15.105:0/2505335246 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fa04c06c530 msgr2=0x7fa04c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:14.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.316+0000 7fa068641700 1 --2- 172.21.15.105:0/2505335246 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fa04c06c530 0x7fa04c06e9f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa05c007fd0 tx=0x7fa05c007480 comp rx=0 tx=0).stop 2024-04-06T02:50:14.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.317+0000 7fa068641700 1 -- 172.21.15.105:0/2505335246 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa0601959f0 msgr2=0x7fa06019aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:14.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.317+0000 7fa068641700 1 --2- 172.21.15.105:0/2505335246 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa0601959f0 0x7fa06019aa60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa050008000 tx=0x7fa0500048c0 comp rx=0 tx=0).stop 2024-04-06T02:50:14.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.317+0000 7fa068641700 1 -- 172.21.15.105:0/2505335246 shutdown_connections 2024-04-06T02:50:14.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.317+0000 7fa068641700 1 --2- 172.21.15.105:0/2505335246 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fa04c06c530 0x7fa04c06e9f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:14.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.317+0000 7fa068641700 1 --2- 172.21.15.105:0/2505335246 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa0600ffab0 0x7fa0601954b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:14.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.317+0000 7fa068641700 1 --2- 172.21.15.105:0/2505335246 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa0601959f0 0x7fa06019aa60 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:14.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.317+0000 7fa068641700 1 -- 172.21.15.105:0/2505335246 >> 172.21.15.105:0/2505335246 conn(0x7fa0600fb030 msgr2=0x7fa060103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:50:14.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.317+0000 7fa068641700 1 -- 172.21.15.105:0/2505335246 shutdown_connections 2024-04-06T02:50:14.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.317+0000 7fa068641700 1 -- 172.21.15.105:0/2505335246 wait complete. 2024-04-06T02:50:14.465 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.479+0000 7fd402a25700 1 -- 172.21.15.105:0/2572872221 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd3fc100440 msgr2=0x7fd3fc1008a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:14.465 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.479+0000 7fd402a25700 1 --2- 172.21.15.105:0/2572872221 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd3fc100440 0x7fd3fc1008a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0009910 tx=0x7fd3f0009c20 comp rx=0 tx=0).stop 2024-04-06T02:50:14.465 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.479+0000 7fd402a25700 1 -- 172.21.15.105:0/2572872221 shutdown_connections 2024-04-06T02:50:14.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.479+0000 7fd402a25700 1 --2- 172.21.15.105:0/2572872221 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd3fc100440 0x7fd3fc1008a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:14.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.479+0000 7fd402a25700 1 --2- 172.21.15.105:0/2572872221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd3fc0ff240 0x7fd3fc0ff660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:14.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.479+0000 7fd402a25700 1 -- 172.21.15.105:0/2572872221 >> 172.21.15.105:0/2572872221 conn(0x7fd3fc0fa7c0 msgr2=0x7fd3fc0fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:50:14.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.480+0000 7fd402a25700 1 -- 172.21.15.105:0/2572872221 shutdown_connections 2024-04-06T02:50:14.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.480+0000 7fd402a25700 1 -- 172.21.15.105:0/2572872221 wait complete. 2024-04-06T02:50:14.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.481+0000 7fd402a25700 1 Processor -- start 2024-04-06T02:50:14.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.481+0000 7fd402a25700 1 -- start start 2024-04-06T02:50:14.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.481+0000 7fd402a25700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd3fc0ff240 0x7fd3fc1031d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:14.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.481+0000 7fd402a25700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd3fc100440 0x7fd3fc101820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:14.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.481+0000 7fd402a25700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc101d60 con 0x7fd3fc100440 2024-04-06T02:50:14.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.481+0000 7fd402a25700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc101ea0 con 0x7fd3fc0ff240 2024-04-06T02:50:14.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.482+0000 7fd3fbfff700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd3fc0ff240 0x7fd3fc1031d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:14.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.482+0000 7fd3fbfff700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd3fc0ff240 0x7fd3fc1031d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.105:51492/0 (socket says 172.21.15.105:51492) 2024-04-06T02:50:14.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.482+0000 7fd3fbfff700 1 -- 172.21.15.105:0/2156385166 learned_addr learned my addr 172.21.15.105:0/2156385166 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:50:14.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.482+0000 7fd3fb7fe700 1 --2- 172.21.15.105:0/2156385166 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd3fc100440 0x7fd3fc101820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:14.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.482+0000 7fd3fbfff700 1 -- 172.21.15.105:0/2156385166 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd3fc100440 msgr2=0x7fd3fc101820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:14.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.482+0000 7fd3fbfff700 1 --2- 172.21.15.105:0/2156385166 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd3fc100440 0x7fd3fc101820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:14.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.482+0000 7fd3fbfff700 1 -- 172.21.15.105:0/2156385166 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ec009520 con 0x7fd3fc0ff240 2024-04-06T02:50:14.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.482+0000 7fd3fb7fe700 1 --2- 172.21.15.105:0/2156385166 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd3fc100440 0x7fd3fc101820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T02:50:14.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.482+0000 7fd3fbfff700 1 --2- 172.21.15.105:0/2156385166 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd3fc0ff240 0x7fd3fc1031d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec00e9d0 tx=0x7fd3ec00ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:50:14.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.482+0000 7fd3f97fa700 1 -- 172.21.15.105:0/2156385166 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3ec00cba0 con 0x7fd3fc0ff240 2024-04-06T02:50:14.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.483+0000 7fd3f97fa700 1 -- 172.21.15.105:0/2156385166 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd3ec00cd00 con 0x7fd3fc0ff240 2024-04-06T02:50:14.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.483+0000 7fd3f97fa700 1 -- 172.21.15.105:0/2156385166 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3ec010430 con 0x7fd3fc0ff240 2024-04-06T02:50:14.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.483+0000 7fd402a25700 1 -- 172.21.15.105:0/2156385166 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f00095f0 con 0x7fd3fc0ff240 2024-04-06T02:50:14.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.483+0000 7fd402a25700 1 -- 172.21.15.105:0/2156385166 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3fc1024e0 con 0x7fd3fc0ff240 2024-04-06T02:50:14.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.484+0000 7fd402a25700 1 -- 172.21.15.105:0/2156385166 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3fc18fbc0 con 0x7fd3fc0ff240 2024-04-06T02:50:14.470 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.485+0000 7fd3f97fa700 1 -- 172.21.15.105:0/2156385166 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fd3ec004750 con 0x7fd3fc0ff240 2024-04-06T02:50:14.471 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.485+0000 7fd3f97fa700 1 --2- 172.21.15.105:0/2156385166 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fd3e406c4e0 0x7fd3e406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:14.471 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.486+0000 7fd3f97fa700 1 -- 172.21.15.105:0/2156385166 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fd3ec014070 con 0x7fd3fc0ff240 2024-04-06T02:50:14.471 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.486+0000 7fd3fb7fe700 1 --2- 172.21.15.105:0/2156385166 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fd3e406c4e0 0x7fd3e406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:14.471 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.486+0000 7fd3fb7fe700 1 --2- 172.21.15.105:0/2156385166 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fd3e406c4e0 0x7fd3e406e9a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd3f000b6a0 tx=0x7fd3f0005fb0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:50:14.480 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.494+0000 7fd3f97fa700 1 -- 172.21.15.105:0/2156385166 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd3ec059e40 con 0x7fd3fc0ff240 2024-04-06T02:50:14.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.899+0000 7fd402a25700 1 -- 172.21.15.105:0/2156385166 --> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3fc0611d0 con 0x7fd3e406c4e0 2024-04-06T02:50:14.886 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:14 smithi105 ceph-mon[29228]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:50:14.903 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.918+0000 7fd3f97fa700 1 -- 172.21.15.105:0/2156385166 <== mgr.14221 v2:172.21.15.105:6800/2334550011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3480 (secure 0 0 0) 0x7fd3fc0611d0 con 0x7fd3e406c4e0 2024-04-06T02:50:14.903 INFO:teuthology.orchestra.run.smithi105.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T02:50:14.903 INFO:teuthology.orchestra.run.smithi105.stdout:alertmanager.smithi105 smithi105 *:9093,9094 running (5m) 62s ago 7m 25.4M - 0.25.0 c8568f914cd2 5b927ee87a96 2024-04-06T02:50:14.904 INFO:teuthology.orchestra.run.smithi105.stdout:ceph-exporter.smithi105 smithi105 running (7m) 62s ago 7m 7453k - 18.2.1 5be31c24972a 448b72643fea 2024-04-06T02:50:14.904 INFO:teuthology.orchestra.run.smithi105.stdout:ceph-exporter.smithi184 smithi184 running (6m) 67s ago 6m 7377k - 18.2.1 5be31c24972a 52fd408df95f 2024-04-06T02:50:14.904 INFO:teuthology.orchestra.run.smithi105.stdout:crash.smithi105 smithi105 running (7m) 62s ago 7m 6660k - 18.2.1 5be31c24972a 8e21a7fab0b2 2024-04-06T02:50:14.904 INFO:teuthology.orchestra.run.smithi105.stdout:crash.smithi184 smithi184 running (6m) 67s ago 6m 6660k - 18.2.1 5be31c24972a a0c065bc6908 2024-04-06T02:50:14.904 INFO:teuthology.orchestra.run.smithi105.stdout:grafana.smithi105 smithi105 *:3000 running (5m) 62s ago 6m 81.1M - 9.4.7 954c08fa6188 413a7bf83436 2024-04-06T02:50:14.904 INFO:teuthology.orchestra.run.smithi105.stdout:mds.cephfs.smithi105.mrivth smithi105 running (88s) 62s ago 87s 11.3M - 18.2.1 5be31c24972a c682a8e71935 2024-04-06T02:50:14.904 INFO:teuthology.orchestra.run.smithi105.stdout:mds.cephfs.smithi105.oxxcbq smithi105 running (81s) 62s ago 81s 18.2M - 18.2.1 5be31c24972a 9ca38954f88a 2024-04-06T02:50:14.904 INFO:teuthology.orchestra.run.smithi105.stdout:mds.cephfs.smithi184.htgrjn smithi184 running (78s) 67s ago 78s 15.3M - 18.2.1 5be31c24972a d6049e6eb483 2024-04-06T02:50:14.904 INFO:teuthology.orchestra.run.smithi105.stdout:mds.cephfs.smithi184.oochxp smithi184 running (85s) 67s ago 85s 16.0M - 18.2.1 5be31c24972a fd16feedab03 2024-04-06T02:50:14.904 INFO:teuthology.orchestra.run.smithi105.stdout:mgr.smithi105.vqqcpx smithi105 *:9283,8765,8443 running (8m) 62s ago 8m 489M - 18.2.1 5be31c24972a 07c0a110a1b8 2024-04-06T02:50:14.904 INFO:teuthology.orchestra.run.smithi105.stdout:mgr.smithi184.fzhhql smithi184 *:8443,9283,8765 running (5m) 67s ago 5m 443M - 18.2.1 5be31c24972a a09137f55e51 2024-04-06T02:50:14.904 INFO:teuthology.orchestra.run.smithi105.stdout:mon.smithi105 smithi105 running (8m) 62s ago 8m 46.7M 2048M 18.2.1 5be31c24972a d27bf0498159 2024-04-06T02:50:14.904 INFO:teuthology.orchestra.run.smithi105.stdout:mon.smithi184 smithi184 running (5m) 67s ago 5m 38.9M 2048M 18.2.1 5be31c24972a 9054b877f58f 2024-04-06T02:50:14.904 INFO:teuthology.orchestra.run.smithi105.stdout:node-exporter.smithi105 smithi105 *:9100 running (7m) 62s ago 7m 15.9M - 1.5.0 0da6a335fe13 543e8799de93 2024-04-06T02:50:14.904 INFO:teuthology.orchestra.run.smithi105.stdout:node-exporter.smithi184 smithi184 *:9100 running (6m) 67s ago 6m 15.7M - 1.5.0 0da6a335fe13 7688716a6f4a 2024-04-06T02:50:14.905 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0 smithi105 running (5m) 62s ago 5m 45.9M 3946M 18.2.1 5be31c24972a 339c3f865ef9 2024-04-06T02:50:14.905 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1 smithi105 running (4m) 62s ago 4m 48.5M 3946M 18.2.1 5be31c24972a eb4cf9ea6b1b 2024-04-06T02:50:14.905 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2 smithi105 running (4m) 62s ago 4m 45.8M 3946M 18.2.1 5be31c24972a 0392f09bdb6b 2024-04-06T02:50:14.905 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3 smithi184 running (3m) 67s ago 3m 47.2M 4970M 18.2.1 5be31c24972a e833150f2893 2024-04-06T02:50:14.905 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4 smithi184 running (3m) 67s ago 3m 44.2M 4970M 18.2.1 5be31c24972a a13051e8d0fb 2024-04-06T02:50:14.905 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5 smithi184 running (2m) 67s ago 2m 46.9M 4970M 18.2.1 5be31c24972a b7b1ce8932de 2024-04-06T02:50:14.905 INFO:teuthology.orchestra.run.smithi105.stdout:prometheus.smithi105 smithi105 *:9095 running (5m) 62s ago 6m 52.7M - 2.43.0 a07b618ecd1d ee5967404456 2024-04-06T02:50:14.906 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.920+0000 7fd402a25700 1 -- 172.21.15.105:0/2156385166 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fd3e406c4e0 msgr2=0x7fd3e406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:14.906 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.920+0000 7fd402a25700 1 --2- 172.21.15.105:0/2156385166 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fd3e406c4e0 0x7fd3e406e9a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd3f000b6a0 tx=0x7fd3f0005fb0 comp rx=0 tx=0).stop 2024-04-06T02:50:14.906 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.920+0000 7fd402a25700 1 -- 172.21.15.105:0/2156385166 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd3fc0ff240 msgr2=0x7fd3fc1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:14.906 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.920+0000 7fd402a25700 1 --2- 172.21.15.105:0/2156385166 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd3fc0ff240 0x7fd3fc1031d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec00e9d0 tx=0x7fd3ec00ece0 comp rx=0 tx=0).stop 2024-04-06T02:50:14.906 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.920+0000 7fd402a25700 1 -- 172.21.15.105:0/2156385166 shutdown_connections 2024-04-06T02:50:14.906 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.920+0000 7fd402a25700 1 --2- 172.21.15.105:0/2156385166 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fd3e406c4e0 0x7fd3e406e9a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:14.906 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.920+0000 7fd402a25700 1 --2- 172.21.15.105:0/2156385166 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd3fc0ff240 0x7fd3fc1031d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:14.906 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.920+0000 7fd402a25700 1 --2- 172.21.15.105:0/2156385166 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd3fc100440 0x7fd3fc101820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:14.906 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.920+0000 7fd402a25700 1 -- 172.21.15.105:0/2156385166 >> 172.21.15.105:0/2156385166 conn(0x7fd3fc0fa7c0 msgr2=0x7fd3fc0fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:50:14.906 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.920+0000 7fd402a25700 1 -- 172.21.15.105:0/2156385166 shutdown_connections 2024-04-06T02:50:14.906 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:14.920+0000 7fd402a25700 1 -- 172.21.15.105:0/2156385166 wait complete. 2024-04-06T02:50:14.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:14 smithi184 ceph-mon[37439]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:50:15.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.044+0000 7fd57dcc5700 1 -- 172.21.15.105:0/1553467477 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd578100bc0 msgr2=0x7fd578100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:15.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.044+0000 7fd57dcc5700 1 --2- 172.21.15.105:0/1553467477 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd578100bc0 0x7fd578100fe0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd568009910 tx=0x7fd568009c20 comp rx=0 tx=0).stop 2024-04-06T02:50:15.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.044+0000 7fd57dcc5700 1 -- 172.21.15.105:0/1553467477 shutdown_connections 2024-04-06T02:50:15.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.044+0000 7fd57dcc5700 1 --2- 172.21.15.105:0/1553467477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd578101dc0 0x7fd578102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:15.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.044+0000 7fd57dcc5700 1 --2- 172.21.15.105:0/1553467477 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd578100bc0 0x7fd578100fe0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:15.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.044+0000 7fd57dcc5700 1 -- 172.21.15.105:0/1553467477 >> 172.21.15.105:0/1553467477 conn(0x7fd5780fc140 msgr2=0x7fd5780fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:50:15.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.044+0000 7fd57dcc5700 1 -- 172.21.15.105:0/1553467477 shutdown_connections 2024-04-06T02:50:15.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.044+0000 7fd57dcc5700 1 -- 172.21.15.105:0/1553467477 wait complete. 2024-04-06T02:50:15.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.045+0000 7fd57dcc5700 1 Processor -- start 2024-04-06T02:50:15.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.045+0000 7fd57dcc5700 1 -- start start 2024-04-06T02:50:15.032 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.045+0000 7fd57dcc5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd578100bc0 0x7fd578074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:15.032 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.045+0000 7fd57dcc5700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd578101dc0 0x7fd578073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:15.032 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.045+0000 7fd57dcc5700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5780736c0 con 0x7fd578100bc0 2024-04-06T02:50:15.032 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.045+0000 7fd57dcc5700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd578073800 con 0x7fd578101dc0 2024-04-06T02:50:15.032 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.045+0000 7fd576ffd700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd578101dc0 0x7fd578073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:15.032 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.045+0000 7fd576ffd700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd578101dc0 0x7fd578073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.105:51506/0 (socket says 172.21.15.105:51506) 2024-04-06T02:50:15.032 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.046+0000 7fd576ffd700 1 -- 172.21.15.105:0/2553042632 learned_addr learned my addr 172.21.15.105:0/2553042632 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:50:15.032 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.046+0000 7fd5777fe700 1 --2- 172.21.15.105:0/2553042632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd578100bc0 0x7fd578074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:15.032 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.046+0000 7fd576ffd700 1 -- 172.21.15.105:0/2553042632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd578100bc0 msgr2=0x7fd578074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:15.032 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.046+0000 7fd576ffd700 1 --2- 172.21.15.105:0/2553042632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd578100bc0 0x7fd578074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:15.032 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.046+0000 7fd576ffd700 1 -- 172.21.15.105:0/2553042632 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5680095f0 con 0x7fd578101dc0 2024-04-06T02:50:15.032 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.046+0000 7fd576ffd700 1 --2- 172.21.15.105:0/2553042632 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd578101dc0 0x7fd578073180 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd56c00e920 tx=0x7fd56c00ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:50:15.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.046+0000 7fd574ff9700 1 -- 172.21.15.105:0/2553042632 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd56c009790 con 0x7fd578101dc0 2024-04-06T02:50:15.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.046+0000 7fd57dcc5700 1 -- 172.21.15.105:0/2553042632 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd578073ae0 con 0x7fd578101dc0 2024-04-06T02:50:15.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.046+0000 7fd574ff9700 1 -- 172.21.15.105:0/2553042632 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd56c00cba0 con 0x7fd578101dc0 2024-04-06T02:50:15.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.046+0000 7fd57dcc5700 1 -- 172.21.15.105:0/2553042632 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd57819e3d0 con 0x7fd578101dc0 2024-04-06T02:50:15.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.046+0000 7fd574ff9700 1 -- 172.21.15.105:0/2553042632 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd56c018630 con 0x7fd578101dc0 2024-04-06T02:50:15.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.047+0000 7fd57dcc5700 1 -- 172.21.15.105:0/2553042632 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd578109720 con 0x7fd578101dc0 2024-04-06T02:50:15.034 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.049+0000 7fd574ff9700 1 -- 172.21.15.105:0/2553042632 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fd56c010430 con 0x7fd578101dc0 2024-04-06T02:50:15.035 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.049+0000 7fd574ff9700 1 --2- 172.21.15.105:0/2553042632 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fd56006c4e0 0x7fd56006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:15.035 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.049+0000 7fd574ff9700 1 -- 172.21.15.105:0/2553042632 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fd56c014070 con 0x7fd578101dc0 2024-04-06T02:50:15.035 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.049+0000 7fd5777fe700 1 --2- 172.21.15.105:0/2553042632 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fd56006c4e0 0x7fd56006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:15.035 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.050+0000 7fd5777fe700 1 --2- 172.21.15.105:0/2553042632 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fd56006c4e0 0x7fd56006e9a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd568009fd0 tx=0x7fd568005940 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:50:15.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.058+0000 7fd574ff9700 1 -- 172.21.15.105:0/2553042632 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd56c0520e0 con 0x7fd578101dc0 2024-04-06T02:50:15.333 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.347+0000 7fd57dcc5700 1 -- 172.21.15.105:0/2553042632 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd57819e980 con 0x7fd578101dc0 2024-04-06T02:50:15.334 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.348+0000 7fd574ff9700 1 -- 172.21.15.105:0/2553042632 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7fd56c059ba0 con 0x7fd578101dc0 2024-04-06T02:50:15.334 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-06T02:50:15.334 INFO:teuthology.orchestra.run.smithi105.stdout: "mon": { 2024-04-06T02:50:15.334 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-06T02:50:15.334 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-06T02:50:15.334 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr": { 2024-04-06T02:50:15.334 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-06T02:50:15.334 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-06T02:50:15.334 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": { 2024-04-06T02:50:15.335 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-06T02:50:15.335 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-06T02:50:15.335 INFO:teuthology.orchestra.run.smithi105.stdout: "mds": { 2024-04-06T02:50:15.335 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-06T02:50:15.335 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-06T02:50:15.335 INFO:teuthology.orchestra.run.smithi105.stdout: "overall": { 2024-04-06T02:50:15.335 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 14 2024-04-06T02:50:15.335 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-06T02:50:15.335 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-06T02:50:15.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.351+0000 7fd57dcc5700 1 -- 172.21.15.105:0/2553042632 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fd56006c4e0 msgr2=0x7fd56006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:15.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.351+0000 7fd57dcc5700 1 --2- 172.21.15.105:0/2553042632 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fd56006c4e0 0x7fd56006e9a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd568009fd0 tx=0x7fd568005940 comp rx=0 tx=0).stop 2024-04-06T02:50:15.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.351+0000 7fd57dcc5700 1 -- 172.21.15.105:0/2553042632 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd578101dc0 msgr2=0x7fd578073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:15.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.351+0000 7fd57dcc5700 1 --2- 172.21.15.105:0/2553042632 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd578101dc0 0x7fd578073180 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd56c00e920 tx=0x7fd56c00ece0 comp rx=0 tx=0).stop 2024-04-06T02:50:15.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.351+0000 7fd57dcc5700 1 -- 172.21.15.105:0/2553042632 shutdown_connections 2024-04-06T02:50:15.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.351+0000 7fd57dcc5700 1 --2- 172.21.15.105:0/2553042632 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fd56006c4e0 0x7fd56006e9a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:15.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.351+0000 7fd57dcc5700 1 --2- 172.21.15.105:0/2553042632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd578100bc0 0x7fd578074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:15.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.351+0000 7fd57dcc5700 1 --2- 172.21.15.105:0/2553042632 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd578101dc0 0x7fd578073180 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:15.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.351+0000 7fd57dcc5700 1 -- 172.21.15.105:0/2553042632 >> 172.21.15.105:0/2553042632 conn(0x7fd5780fc140 msgr2=0x7fd578104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:50:15.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.351+0000 7fd57dcc5700 1 -- 172.21.15.105:0/2553042632 shutdown_connections 2024-04-06T02:50:15.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.351+0000 7fd57dcc5700 1 -- 172.21.15.105:0/2553042632 wait complete. 2024-04-06T02:50:15.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.466+0000 7ff0180f2700 1 -- 172.21.15.105:0/853337849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff010100cb0 msgr2=0x7ff010101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:15.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.466+0000 7ff0180f2700 1 --2- 172.21.15.105:0/853337849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff010100cb0 0x7ff010101110 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff000009910 tx=0x7ff000009c20 comp rx=0 tx=0).stop 2024-04-06T02:50:15.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.466+0000 7ff0180f2700 1 -- 172.21.15.105:0/853337849 shutdown_connections 2024-04-06T02:50:15.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.466+0000 7ff0180f2700 1 --2- 172.21.15.105:0/853337849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff010100cb0 0x7ff010101110 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:15.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.466+0000 7ff0180f2700 1 --2- 172.21.15.105:0/853337849 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0100ffab0 0x7ff0100ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:15.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.466+0000 7ff0180f2700 1 -- 172.21.15.105:0/853337849 >> 172.21.15.105:0/853337849 conn(0x7ff0100fb050 msgr2=0x7ff0100fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:50:15.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.466+0000 7ff0180f2700 1 -- 172.21.15.105:0/853337849 shutdown_connections 2024-04-06T02:50:15.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.466+0000 7ff0180f2700 1 -- 172.21.15.105:0/853337849 wait complete. 2024-04-06T02:50:15.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.467+0000 7ff0180f2700 1 Processor -- start 2024-04-06T02:50:15.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.467+0000 7ff0180f2700 1 -- start start 2024-04-06T02:50:15.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.467+0000 7ff0180f2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0100ffab0 0x7ff010196360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:15.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.467+0000 7ff0180f2700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff010100cb0 0x7ff0101968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:15.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.467+0000 7ff0180f2700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff010196ec0 con 0x7ff0100ffab0 2024-04-06T02:50:15.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.467+0000 7ff0180f2700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff010197000 con 0x7ff010100cb0 2024-04-06T02:50:15.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.467+0000 7ff015e8e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0100ffab0 0x7ff010196360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:15.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.467+0000 7ff015e8e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0100ffab0 0x7ff010196360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:58430/0 (socket says 172.21.15.105:58430) 2024-04-06T02:50:15.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.467+0000 7ff015e8e700 1 -- 172.21.15.105:0/3819961937 learned_addr learned my addr 172.21.15.105:0/3819961937 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:50:15.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.467+0000 7ff01568d700 1 --2- 172.21.15.105:0/3819961937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff010100cb0 0x7ff0101968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:15.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.467+0000 7ff015e8e700 1 -- 172.21.15.105:0/3819961937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff010100cb0 msgr2=0x7ff0101968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:15.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.467+0000 7ff015e8e700 1 --2- 172.21.15.105:0/3819961937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff010100cb0 0x7ff0101968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:15.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.467+0000 7ff015e8e700 1 -- 172.21.15.105:0/3819961937 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0000095f0 con 0x7ff0100ffab0 2024-04-06T02:50:15.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.468+0000 7ff01568d700 1 --2- 172.21.15.105:0/3819961937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff010100cb0 0x7ff0101968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T02:50:15.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.468+0000 7ff015e8e700 1 --2- 172.21.15.105:0/3819961937 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0100ffab0 0x7ff010196360 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff00c00e8f0 tx=0x7ff00c00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:50:15.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.468+0000 7ff006ffd700 1 -- 172.21.15.105:0/3819961937 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff00c009790 con 0x7ff0100ffab0 2024-04-06T02:50:15.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.468+0000 7ff006ffd700 1 -- 172.21.15.105:0/3819961937 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff00c00c810 con 0x7ff0100ffab0 2024-04-06T02:50:15.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.468+0000 7ff0180f2700 1 -- 172.21.15.105:0/3819961937 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff01019bab0 con 0x7ff0100ffab0 2024-04-06T02:50:15.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.468+0000 7ff006ffd700 1 -- 172.21.15.105:0/3819961937 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff00c003d20 con 0x7ff0100ffab0 2024-04-06T02:50:15.456 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.468+0000 7ff0180f2700 1 -- 172.21.15.105:0/3819961937 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff01019c000 con 0x7ff0100ffab0 2024-04-06T02:50:15.456 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.469+0000 7ff0180f2700 1 -- 172.21.15.105:0/3819961937 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff010109710 con 0x7ff0100ffab0 2024-04-06T02:50:15.456 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.470+0000 7ff006ffd700 1 -- 172.21.15.105:0/3819961937 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7ff00c0041f0 con 0x7ff0100ffab0 2024-04-06T02:50:15.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.471+0000 7ff006ffd700 1 --2- 172.21.15.105:0/3819961937 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7feffc06c4e0 0x7feffc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:15.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.471+0000 7ff01568d700 1 --2- 172.21.15.105:0/3819961937 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7feffc06c4e0 0x7feffc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:15.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.472+0000 7ff006ffd700 1 -- 172.21.15.105:0/3819961937 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ff00c014070 con 0x7ff0100ffab0 2024-04-06T02:50:15.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.472+0000 7ff01568d700 1 --2- 172.21.15.105:0/3819961937 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7feffc06c4e0 0x7feffc06e9a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff000009fd0 tx=0x7ff000005940 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:50:15.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.476+0000 7ff006ffd700 1 -- 172.21.15.105:0/3819961937 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff00c059d20 con 0x7ff0100ffab0 2024-04-06T02:50:15.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:15 smithi105 ceph-mon[29228]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:50:15.676 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:15 smithi105 ceph-mon[29228]: pgmap v225: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-06T02:50:15.676 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:15 smithi105 ceph-mon[29228]: from='client.24315 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:50:15.676 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:15 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:50:15.676 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:15 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:50:15.676 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:15 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:50:15.676 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:15 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/2553042632' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:50:15.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.769+0000 7ff0180f2700 1 -- 172.21.15.105:0/3819961937 --> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff010105600 con 0x7feffc06c4e0 2024-04-06T02:50:15.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.771+0000 7ff006ffd700 1 -- 172.21.15.105:0/3819961937 <== mgr.14221 v2:172.21.15.105:6800/2334550011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7ff010105600 con 0x7feffc06c4e0 2024-04-06T02:50:15.758 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-06T02:50:15.758 INFO:teuthology.orchestra.run.smithi105.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T02:50:15.758 INFO:teuthology.orchestra.run.smithi105.stdout: "in_progress": true, 2024-04-06T02:50:15.758 INFO:teuthology.orchestra.run.smithi105.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-06T02:50:15.758 INFO:teuthology.orchestra.run.smithi105.stdout: "services_complete": [], 2024-04-06T02:50:15.758 INFO:teuthology.orchestra.run.smithi105.stdout: "progress": "0/2 daemons upgraded", 2024-04-06T02:50:15.758 INFO:teuthology.orchestra.run.smithi105.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c image on host smithi184", 2024-04-06T02:50:15.759 INFO:teuthology.orchestra.run.smithi105.stdout: "is_paused": false 2024-04-06T02:50:15.759 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-06T02:50:15.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.773+0000 7ff0180f2700 1 -- 172.21.15.105:0/3819961937 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7feffc06c4e0 msgr2=0x7feffc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:15.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.773+0000 7ff0180f2700 1 --2- 172.21.15.105:0/3819961937 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7feffc06c4e0 0x7feffc06e9a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff000009fd0 tx=0x7ff000005940 comp rx=0 tx=0).stop 2024-04-06T02:50:15.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.773+0000 7ff0180f2700 1 -- 172.21.15.105:0/3819961937 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0100ffab0 msgr2=0x7ff010196360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:15.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.773+0000 7ff0180f2700 1 --2- 172.21.15.105:0/3819961937 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0100ffab0 0x7ff010196360 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff00c00e8f0 tx=0x7ff00c00ec00 comp rx=0 tx=0).stop 2024-04-06T02:50:15.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.773+0000 7ff0180f2700 1 -- 172.21.15.105:0/3819961937 shutdown_connections 2024-04-06T02:50:15.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.773+0000 7ff0180f2700 1 --2- 172.21.15.105:0/3819961937 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7feffc06c4e0 0x7feffc06e9a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:15.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.773+0000 7ff0180f2700 1 --2- 172.21.15.105:0/3819961937 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0100ffab0 0x7ff010196360 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:15.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.773+0000 7ff0180f2700 1 --2- 172.21.15.105:0/3819961937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff010100cb0 0x7ff0101968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:15.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.773+0000 7ff0180f2700 1 -- 172.21.15.105:0/3819961937 >> 172.21.15.105:0/3819961937 conn(0x7ff0100fb050 msgr2=0x7ff010103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:50:15.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.773+0000 7ff0180f2700 1 -- 172.21.15.105:0/3819961937 shutdown_connections 2024-04-06T02:50:15.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:15.773+0000 7ff0180f2700 1 -- 172.21.15.105:0/3819961937 wait complete. 2024-04-06T02:50:15.882 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:15 smithi184 ceph-mon[37439]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:50:15.882 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:15 smithi184 ceph-mon[37439]: pgmap v225: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-06T02:50:15.882 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:15 smithi184 ceph-mon[37439]: from='client.24315 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:50:15.882 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:15 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:50:15.882 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:15 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T02:50:15.882 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:15 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:50:15.882 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:15 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/2553042632' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:50:16.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:16 smithi105 ceph-mon[29228]: Upgrade: Target is version 19.0.0-2027-g9f192350 (unknown) 2024-04-06T02:50:16.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:16 smithi105 ceph-mon[29228]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c'] 2024-04-06T02:50:16.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:16 smithi105 ceph-mon[29228]: Upgrade: Need to upgrade myself (mgr.smithi105.vqqcpx) 2024-04-06T02:50:16.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:16 smithi105 ceph-mon[29228]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c on smithi184 2024-04-06T02:50:16.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:16 smithi105 ceph-mon[29228]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:50:16.964 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:16 smithi184 ceph-mon[37439]: Upgrade: Target is version 19.0.0-2027-g9f192350 (unknown) 2024-04-06T02:50:16.964 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:16 smithi184 ceph-mon[37439]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c'] 2024-04-06T02:50:16.964 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:16 smithi184 ceph-mon[37439]: Upgrade: Need to upgrade myself (mgr.smithi105.vqqcpx) 2024-04-06T02:50:16.964 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:16 smithi184 ceph-mon[37439]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c on smithi184 2024-04-06T02:50:16.964 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:16 smithi184 ceph-mon[37439]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:50:17.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:17 smithi105 ceph-mon[29228]: pgmap v226: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-06T02:50:17.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:17 smithi184 ceph-mon[37439]: pgmap v226: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-06T02:50:19.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:19 smithi184 ceph-mon[37439]: pgmap v227: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-06T02:50:20.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:19 smithi105 ceph-mon[29228]: pgmap v227: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-06T02:50:21.945 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:21 smithi105 ceph-mon[29228]: pgmap v228: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:50:21.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:21 smithi184 ceph-mon[37439]: pgmap v228: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:50:23.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:23 smithi105 ceph-mon[29228]: pgmap v229: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:50:23.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:23 smithi184 ceph-mon[37439]: pgmap v229: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:50:24.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:24 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:50:25.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:24 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:50:25.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:25 smithi184 ceph-mon[37439]: pgmap v230: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:50:26.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:25 smithi105 ceph-mon[29228]: pgmap v230: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:50:27.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:27 smithi184 ceph-mon[37439]: pgmap v231: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:50:28.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:27 smithi105 ceph-mon[29228]: pgmap v231: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:50:29.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:29 smithi184 ceph-mon[37439]: pgmap v232: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:50:30.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:29 smithi105 ceph-mon[29228]: pgmap v232: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:50:31.759 INFO:tasks.workunit.client.1.smithi184.stderr:Updating files: 97% (12120/12461) Updating files: 98% (12212/12461) Updating files: 99% (12337/12461) Updating files: 100% (12461/12461) Updating files: 100% (12461/12461), done. 2024-04-06T02:50:31.836 INFO:tasks.workunit.client.0.smithi105.stderr:Note: switching to '584a67ccb9eb224c5312dccd96a82e67b0645518'. 2024-04-06T02:50:31.836 INFO:tasks.workunit.client.0.smithi105.stderr: 2024-04-06T02:50:31.836 INFO:tasks.workunit.client.0.smithi105.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-06T02:50:31.836 INFO:tasks.workunit.client.0.smithi105.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-06T02:50:31.836 INFO:tasks.workunit.client.0.smithi105.stderr:state without impacting any branches by switching back to a branch. 2024-04-06T02:50:31.836 INFO:tasks.workunit.client.0.smithi105.stderr: 2024-04-06T02:50:31.837 INFO:tasks.workunit.client.0.smithi105.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-06T02:50:31.837 INFO:tasks.workunit.client.0.smithi105.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-06T02:50:31.837 INFO:tasks.workunit.client.0.smithi105.stderr: 2024-04-06T02:50:31.837 INFO:tasks.workunit.client.0.smithi105.stderr: git switch -c 2024-04-06T02:50:31.837 INFO:tasks.workunit.client.0.smithi105.stderr: 2024-04-06T02:50:31.837 INFO:tasks.workunit.client.0.smithi105.stderr:Or undo this operation with: 2024-04-06T02:50:31.837 INFO:tasks.workunit.client.0.smithi105.stderr: 2024-04-06T02:50:31.837 INFO:tasks.workunit.client.0.smithi105.stderr: git switch - 2024-04-06T02:50:31.837 INFO:tasks.workunit.client.0.smithi105.stderr: 2024-04-06T02:50:31.837 INFO:tasks.workunit.client.0.smithi105.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-06T02:50:31.837 INFO:tasks.workunit.client.0.smithi105.stderr: 2024-04-06T02:50:31.837 INFO:tasks.workunit.client.0.smithi105.stderr:HEAD is now at 584a67ccb9e Merge pull request #56706 from afreen23/wip-65208-squid 2024-04-06T02:50:31.840 DEBUG:teuthology.orchestra.run.smithi105:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-06T02:50:31.874 INFO:tasks.workunit.client.0.smithi105.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-06T02:50:31.876 INFO:tasks.workunit.client.0.smithi105.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-06T02:50:31.876 INFO:tasks.workunit.client.0.smithi105.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-06T02:50:31.980 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:31 smithi184 ceph-mon[37439]: pgmap v233: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:50:31.982 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:31 smithi105 ceph-mon[29228]: pgmap v233: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:50:32.510 INFO:tasks.workunit.client.1.smithi184.stderr:Note: switching to '584a67ccb9eb224c5312dccd96a82e67b0645518'. 2024-04-06T02:50:32.510 INFO:tasks.workunit.client.1.smithi184.stderr: 2024-04-06T02:50:32.510 INFO:tasks.workunit.client.1.smithi184.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-06T02:50:32.510 INFO:tasks.workunit.client.1.smithi184.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-06T02:50:32.511 INFO:tasks.workunit.client.1.smithi184.stderr:state without impacting any branches by switching back to a branch. 2024-04-06T02:50:32.511 INFO:tasks.workunit.client.1.smithi184.stderr: 2024-04-06T02:50:32.511 INFO:tasks.workunit.client.1.smithi184.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-06T02:50:32.511 INFO:tasks.workunit.client.1.smithi184.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-06T02:50:32.511 INFO:tasks.workunit.client.1.smithi184.stderr: 2024-04-06T02:50:32.511 INFO:tasks.workunit.client.1.smithi184.stderr: git switch -c 2024-04-06T02:50:32.511 INFO:tasks.workunit.client.1.smithi184.stderr: 2024-04-06T02:50:32.511 INFO:tasks.workunit.client.1.smithi184.stderr:Or undo this operation with: 2024-04-06T02:50:32.511 INFO:tasks.workunit.client.1.smithi184.stderr: 2024-04-06T02:50:32.511 INFO:tasks.workunit.client.1.smithi184.stderr: git switch - 2024-04-06T02:50:32.511 INFO:tasks.workunit.client.1.smithi184.stderr: 2024-04-06T02:50:32.511 INFO:tasks.workunit.client.1.smithi184.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-06T02:50:32.511 INFO:tasks.workunit.client.1.smithi184.stderr: 2024-04-06T02:50:32.511 INFO:tasks.workunit.client.1.smithi184.stderr:HEAD is now at 584a67ccb9e Merge pull request #56706 from afreen23/wip-65208-squid 2024-04-06T02:50:32.514 DEBUG:teuthology.orchestra.run.smithi184:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-04-06T02:50:32.579 INFO:tasks.workunit.client.1.smithi184.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-06T02:50:32.581 INFO:tasks.workunit.client.1.smithi184.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-06T02:50:32.581 INFO:tasks.workunit.client.1.smithi184.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-06T02:50:32.687 INFO:tasks.workunit.client.0.smithi105.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-06T02:50:32.758 INFO:tasks.workunit.client.0.smithi105.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-06T02:50:32.797 INFO:tasks.workunit.client.0.smithi105.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-06T02:50:32.799 INFO:tasks.workunit.client.0.smithi105.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-06T02:50:32.799 INFO:tasks.workunit.client.0.smithi105.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-06T02:50:32.838 INFO:tasks.workunit.client.0.smithi105.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-06T02:50:32.841 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-06T02:50:32.841 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-06T02:50:32.897 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-04-06T02:50:32.899 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-06T02:50:32.899 DEBUG:teuthology.orchestra.run.smithi105:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=584a67ccb9eb224c5312dccd96a82e67b0645518 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-04-06T02:50:32.958 INFO:tasks.workunit.client.0.smithi105.stderr:+ mkdir -p fsstress 2024-04-06T02:50:32.959 INFO:tasks.workunit.client.0.smithi105.stderr:+ pushd fsstress 2024-04-06T02:50:32.959 INFO:tasks.workunit.client.0.smithi105.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-06T02:50:32.960 INFO:tasks.workunit.client.0.smithi105.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-06T02:50:33.379 INFO:tasks.workunit.client.1.smithi184.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-06T02:50:33.458 INFO:tasks.workunit.client.1.smithi184.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-06T02:50:33.479 INFO:tasks.workunit.client.0.smithi105.stderr:+ tar xzf ltp-full.tgz 2024-04-06T02:50:33.500 INFO:tasks.workunit.client.1.smithi184.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-06T02:50:33.502 INFO:tasks.workunit.client.1.smithi184.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-06T02:50:33.502 INFO:tasks.workunit.client.1.smithi184.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-06T02:50:33.544 INFO:tasks.workunit.client.1.smithi184.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-06T02:50:33.548 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-04-06T02:50:33.548 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-04-06T02:50:33.604 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-04-06T02:50:33.605 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-06T02:50:33.605 DEBUG:teuthology.orchestra.run.smithi184:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=584a67ccb9eb224c5312dccd96a82e67b0645518 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-04-06T02:50:33.665 INFO:tasks.workunit.client.1.smithi184.stderr:+ mkdir -p fsstress 2024-04-06T02:50:33.668 INFO:tasks.workunit.client.1.smithi184.stderr:+ pushd fsstress 2024-04-06T02:50:33.668 INFO:tasks.workunit.client.1.smithi184.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-06T02:50:33.668 INFO:tasks.workunit.client.1.smithi184.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-06T02:50:33.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:33 smithi105 ceph-mon[29228]: pgmap v234: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:50:33.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:33 smithi184 ceph-mon[37439]: pgmap v234: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:50:34.422 INFO:tasks.workunit.client.1.smithi184.stderr:+ tar xzf ltp-full.tgz 2024-04-06T02:50:35.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:35 smithi184 ceph-mon[37439]: pgmap v235: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:50:36.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:35 smithi105 ceph-mon[29228]: pgmap v235: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-06T02:50:37.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:37 smithi184 ceph-mon[37439]: pgmap v236: 65 pgs: 65 active+clean; 2.7 MiB data, 171 MiB used, 536 GiB / 536 GiB avail; 184 KiB/s wr, 0 op/s 2024-04-06T02:50:38.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:37 smithi105 ceph-mon[29228]: pgmap v236: 65 pgs: 65 active+clean; 2.7 MiB data, 171 MiB used, 536 GiB / 536 GiB avail; 184 KiB/s wr, 0 op/s 2024-04-06T02:50:39.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:39 smithi184 ceph-mon[37439]: pgmap v237: 65 pgs: 65 active+clean; 6.7 MiB data, 175 MiB used, 536 GiB / 536 GiB avail; 527 KiB/s wr, 1 op/s 2024-04-06T02:50:39.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:39 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:50:40.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:39 smithi105 ceph-mon[29228]: pgmap v237: 65 pgs: 65 active+clean; 6.7 MiB data, 175 MiB used, 536 GiB / 536 GiB avail; 527 KiB/s wr, 1 op/s 2024-04-06T02:50:40.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:39 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:50:41.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:41 smithi184 ceph-mon[37439]: pgmap v238: 65 pgs: 65 active+clean; 32 MiB data, 270 MiB used, 536 GiB / 536 GiB avail; 2.7 MiB/s wr, 216 op/s 2024-04-06T02:50:42.000 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:41 smithi105 ceph-mon[29228]: pgmap v238: 65 pgs: 65 active+clean; 32 MiB data, 270 MiB used, 536 GiB / 536 GiB avail; 2.7 MiB/s wr, 216 op/s 2024-04-06T02:50:43.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:43 smithi105 ceph-mon[29228]: pgmap v239: 65 pgs: 65 active+clean; 59 MiB data, 370 MiB used, 536 GiB / 536 GiB avail; 5.1 MiB/s wr, 450 op/s 2024-04-06T02:50:43.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:43 smithi184 ceph-mon[37439]: pgmap v239: 65 pgs: 65 active+clean; 59 MiB data, 370 MiB used, 536 GiB / 536 GiB avail; 5.1 MiB/s wr, 450 op/s 2024-04-06T02:50:45.795 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:45 smithi184 ceph-mon[37439]: fsmap cephfs:2 {0=cephfs.smithi105.oxxcbq=up:active,1=cephfs.smithi184.oochxp=up:active} 2 up:standby 2024-04-06T02:50:45.795 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:45 smithi184 ceph-mon[37439]: pgmap v240: 65 pgs: 65 active+clean; 59 MiB data, 370 MiB used, 536 GiB / 536 GiB avail; 5.1 MiB/s wr, 450 op/s 2024-04-06T02:50:45.877 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:45 smithi105 ceph-mon[29228]: fsmap cephfs:2 {0=cephfs.smithi105.oxxcbq=up:active,1=cephfs.smithi184.oochxp=up:active} 2 up:standby 2024-04-06T02:50:45.877 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:45 smithi105 ceph-mon[29228]: pgmap v240: 65 pgs: 65 active+clean; 59 MiB data, 370 MiB used, 536 GiB / 536 GiB avail; 5.1 MiB/s wr, 450 op/s 2024-04-06T02:50:45.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.890+0000 7ffb40102700 1 -- 172.21.15.105:0/1965711800 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffb380ffab0 msgr2=0x7ffb380ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:45.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.890+0000 7ffb40102700 1 --2- 172.21.15.105:0/1965711800 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffb380ffab0 0x7ffb380ffed0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ffb34009910 tx=0x7ffb34009c20 comp rx=0 tx=0).stop 2024-04-06T02:50:45.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.890+0000 7ffb40102700 1 -- 172.21.15.105:0/1965711800 shutdown_connections 2024-04-06T02:50:45.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.890+0000 7ffb40102700 1 --2- 172.21.15.105:0/1965711800 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb38100cb0 0x7ffb38101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:45.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.890+0000 7ffb40102700 1 --2- 172.21.15.105:0/1965711800 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffb380ffab0 0x7ffb380ffed0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:45.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.890+0000 7ffb40102700 1 -- 172.21.15.105:0/1965711800 >> 172.21.15.105:0/1965711800 conn(0x7ffb380fb030 msgr2=0x7ffb380fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:50:45.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.891+0000 7ffb40102700 1 -- 172.21.15.105:0/1965711800 shutdown_connections 2024-04-06T02:50:45.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.891+0000 7ffb40102700 1 -- 172.21.15.105:0/1965711800 wait complete. 2024-04-06T02:50:45.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.891+0000 7ffb40102700 1 Processor -- start 2024-04-06T02:50:45.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.891+0000 7ffb40102700 1 -- start start 2024-04-06T02:50:45.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.892+0000 7ffb40102700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb380ffab0 0x7ffb3819c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:45.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.892+0000 7ffb40102700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffb38100cb0 0x7ffb3819ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:45.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.892+0000 7ffb40102700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3819d4a0 con 0x7ffb38100cb0 2024-04-06T02:50:45.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.892+0000 7ffb40102700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3819d5e0 con 0x7ffb380ffab0 2024-04-06T02:50:45.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.892+0000 7ffb3d69d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffb38100cb0 0x7ffb3819ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:45.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.892+0000 7ffb3d69d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffb38100cb0 0x7ffb3819ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40352/0 (socket says 172.21.15.105:40352) 2024-04-06T02:50:45.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.892+0000 7ffb3d69d700 1 -- 172.21.15.105:0/4058689114 learned_addr learned my addr 172.21.15.105:0/4058689114 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:50:45.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.892+0000 7ffb3de9e700 1 --2- 172.21.15.105:0/4058689114 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb380ffab0 0x7ffb3819c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:45.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.892+0000 7ffb3d69d700 1 -- 172.21.15.105:0/4058689114 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb380ffab0 msgr2=0x7ffb3819c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:45.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.892+0000 7ffb3d69d700 1 --2- 172.21.15.105:0/4058689114 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb380ffab0 0x7ffb3819c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:45.881 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.892+0000 7ffb3d69d700 1 -- 172.21.15.105:0/4058689114 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb340095f0 con 0x7ffb38100cb0 2024-04-06T02:50:45.881 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.892+0000 7ffb3de9e700 1 --2- 172.21.15.105:0/4058689114 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb380ffab0 0x7ffb3819c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T02:50:45.881 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.892+0000 7ffb3d69d700 1 --2- 172.21.15.105:0/4058689114 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffb38100cb0 0x7ffb3819ce80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ffb2800e920 tx=0x7ffb2800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:50:45.881 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.893+0000 7ffb2effd700 1 -- 172.21.15.105:0/4058689114 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb2800c9d0 con 0x7ffb38100cb0 2024-04-06T02:50:45.881 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.893+0000 7ffb40102700 1 -- 172.21.15.105:0/4058689114 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb381a2090 con 0x7ffb38100cb0 2024-04-06T02:50:45.881 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.893+0000 7ffb40102700 1 -- 172.21.15.105:0/4058689114 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb381a25e0 con 0x7ffb38100cb0 2024-04-06T02:50:45.881 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.893+0000 7ffb2effd700 1 -- 172.21.15.105:0/4058689114 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ffb28010430 con 0x7ffb38100cb0 2024-04-06T02:50:45.881 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.893+0000 7ffb2effd700 1 -- 172.21.15.105:0/4058689114 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb28018650 con 0x7ffb38100cb0 2024-04-06T02:50:45.882 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.894+0000 7ffb40102700 1 -- 172.21.15.105:0/4058689114 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb38108610 con 0x7ffb38100cb0 2024-04-06T02:50:45.882 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.895+0000 7ffb2effd700 1 -- 172.21.15.105:0/4058689114 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7ffb280105a0 con 0x7ffb38100cb0 2024-04-06T02:50:45.882 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.895+0000 7ffb2effd700 1 --2- 172.21.15.105:0/4058689114 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7ffb2406c4e0 0x7ffb2406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:45.882 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.896+0000 7ffb2effd700 1 -- 172.21.15.105:0/4058689114 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ffb28014070 con 0x7ffb38100cb0 2024-04-06T02:50:45.882 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.896+0000 7ffb3de9e700 1 --2- 172.21.15.105:0/4058689114 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7ffb2406c4e0 0x7ffb2406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:45.882 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.896+0000 7ffb3de9e700 1 --2- 172.21.15.105:0/4058689114 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7ffb2406c4e0 0x7ffb2406e9a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ffb34009fd0 tx=0x7ffb34005a90 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:50:45.886 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:45.900+0000 7ffb2effd700 1 -- 172.21.15.105:0/4058689114 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ffb280520e0 con 0x7ffb38100cb0 2024-04-06T02:50:46.203 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.218+0000 7ffb40102700 1 -- 172.21.15.105:0/4058689114 --> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb38068b20 con 0x7ffb2406c4e0 2024-04-06T02:50:46.205 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.219+0000 7ffb2effd700 1 -- 172.21.15.105:0/4058689114 <== mgr.14221 v2:172.21.15.105:6800/2334550011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7ffb38068b20 con 0x7ffb2406c4e0 2024-04-06T02:50:46.208 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.223+0000 7ffb40102700 1 -- 172.21.15.105:0/4058689114 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7ffb2406c4e0 msgr2=0x7ffb2406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:46.209 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.223+0000 7ffb40102700 1 --2- 172.21.15.105:0/4058689114 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7ffb2406c4e0 0x7ffb2406e9a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ffb34009fd0 tx=0x7ffb34005a90 comp rx=0 tx=0).stop 2024-04-06T02:50:46.209 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.223+0000 7ffb40102700 1 -- 172.21.15.105:0/4058689114 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffb38100cb0 msgr2=0x7ffb3819ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:46.209 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.223+0000 7ffb40102700 1 --2- 172.21.15.105:0/4058689114 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffb38100cb0 0x7ffb3819ce80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ffb2800e920 tx=0x7ffb2800ec30 comp rx=0 tx=0).stop 2024-04-06T02:50:46.209 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.223+0000 7ffb40102700 1 -- 172.21.15.105:0/4058689114 shutdown_connections 2024-04-06T02:50:46.209 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.223+0000 7ffb40102700 1 --2- 172.21.15.105:0/4058689114 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7ffb2406c4e0 0x7ffb2406e9a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:46.209 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.223+0000 7ffb40102700 1 --2- 172.21.15.105:0/4058689114 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffb380ffab0 0x7ffb3819c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:46.209 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.223+0000 7ffb40102700 1 --2- 172.21.15.105:0/4058689114 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffb38100cb0 0x7ffb3819ce80 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:46.209 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.223+0000 7ffb40102700 1 -- 172.21.15.105:0/4058689114 >> 172.21.15.105:0/4058689114 conn(0x7ffb380fb030 msgr2=0x7ffb38068410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:50:46.209 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.223+0000 7ffb40102700 1 -- 172.21.15.105:0/4058689114 shutdown_connections 2024-04-06T02:50:46.209 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.223+0000 7ffb40102700 1 -- 172.21.15.105:0/4058689114 wait complete. 2024-04-06T02:50:46.222 INFO:teuthology.orchestra.run.smithi105.stdout:true 2024-04-06T02:50:46.320 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.334+0000 7fa825417700 1 -- 172.21.15.105:0/2910904235 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa820101df0 msgr2=0x7fa820102250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:46.320 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.334+0000 7fa825417700 1 --2- 172.21.15.105:0/2910904235 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa820101df0 0x7fa820102250 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa814009910 tx=0x7fa814009c20 comp rx=0 tx=0).stop 2024-04-06T02:50:46.320 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.334+0000 7fa825417700 1 -- 172.21.15.105:0/2910904235 shutdown_connections 2024-04-06T02:50:46.320 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.334+0000 7fa825417700 1 --2- 172.21.15.105:0/2910904235 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa820101df0 0x7fa820102250 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:46.320 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.334+0000 7fa825417700 1 --2- 172.21.15.105:0/2910904235 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa820100bf0 0x7fa820101010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:46.320 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.334+0000 7fa825417700 1 -- 172.21.15.105:0/2910904235 >> 172.21.15.105:0/2910904235 conn(0x7fa8200fc190 msgr2=0x7fa8200fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:50:46.320 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.334+0000 7fa825417700 1 -- 172.21.15.105:0/2910904235 shutdown_connections 2024-04-06T02:50:46.320 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.334+0000 7fa825417700 1 -- 172.21.15.105:0/2910904235 wait complete. 2024-04-06T02:50:46.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.335+0000 7fa825417700 1 Processor -- start 2024-04-06T02:50:46.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.335+0000 7fa825417700 1 -- start start 2024-04-06T02:50:46.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.335+0000 7fa825417700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa820100bf0 0x7fa820196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:46.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.335+0000 7fa825417700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa820101df0 0x7fa8201968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:46.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.335+0000 7fa825417700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa820196e50 con 0x7fa820101df0 2024-04-06T02:50:46.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.335+0000 7fa825417700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa820196f90 con 0x7fa820100bf0 2024-04-06T02:50:46.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.335+0000 7fa81e7fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa820101df0 0x7fa8201968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:46.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.335+0000 7fa81e7fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa820101df0 0x7fa8201968c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40382/0 (socket says 172.21.15.105:40382) 2024-04-06T02:50:46.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.335+0000 7fa81e7fc700 1 -- 172.21.15.105:0/116829925 learned_addr learned my addr 172.21.15.105:0/116829925 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:50:46.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.335+0000 7fa81effd700 1 --2- 172.21.15.105:0/116829925 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa820100bf0 0x7fa820196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:46.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.336+0000 7fa81e7fc700 1 -- 172.21.15.105:0/116829925 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa820100bf0 msgr2=0x7fa820196380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:46.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.336+0000 7fa81e7fc700 1 --2- 172.21.15.105:0/116829925 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa820100bf0 0x7fa820196380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:46.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.336+0000 7fa81e7fc700 1 -- 172.21.15.105:0/116829925 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8140095f0 con 0x7fa820101df0 2024-04-06T02:50:46.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.336+0000 7fa81effd700 1 --2- 172.21.15.105:0/116829925 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa820100bf0 0x7fa820196380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-06T02:50:46.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.336+0000 7fa81e7fc700 1 --2- 172.21.15.105:0/116829925 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa820101df0 0x7fa8201968c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa81400bd70 tx=0x7fa81400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:50:46.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.336+0000 7fa807fff700 1 -- 172.21.15.105:0/116829925 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa81401d070 con 0x7fa820101df0 2024-04-06T02:50:46.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.336+0000 7fa825417700 1 -- 172.21.15.105:0/116829925 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa82019b9f0 con 0x7fa820101df0 2024-04-06T02:50:46.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.336+0000 7fa825417700 1 -- 172.21.15.105:0/116829925 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa82019beb0 con 0x7fa820101df0 2024-04-06T02:50:46.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.336+0000 7fa807fff700 1 -- 172.21.15.105:0/116829925 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa814022470 con 0x7fa820101df0 2024-04-06T02:50:46.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.336+0000 7fa807fff700 1 -- 172.21.15.105:0/116829925 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa81400f670 con 0x7fa820101df0 2024-04-06T02:50:46.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.337+0000 7fa825417700 1 -- 172.21.15.105:0/116829925 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa80c005320 con 0x7fa820101df0 2024-04-06T02:50:46.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.338+0000 7fa807fff700 1 -- 172.21.15.105:0/116829925 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fa814004b70 con 0x7fa820101df0 2024-04-06T02:50:46.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.339+0000 7fa807fff700 1 --2- 172.21.15.105:0/116829925 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fa80806c4e0 0x7fa80806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:46.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.339+0000 7fa807fff700 1 -- 172.21.15.105:0/116829925 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fa81408ce70 con 0x7fa820101df0 2024-04-06T02:50:46.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.339+0000 7fa81effd700 1 --2- 172.21.15.105:0/116829925 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fa80806c4e0 0x7fa80806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:46.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.339+0000 7fa81effd700 1 --2- 172.21.15.105:0/116829925 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fa80806c4e0 0x7fa80806e9a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa820101c50 tx=0x7fa81000a680 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:50:46.333 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.347+0000 7fa807fff700 1 -- 172.21.15.105:0/116829925 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa81405b1b0 con 0x7fa820101df0 2024-04-06T02:50:46.558 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.573+0000 7fa825417700 1 -- 172.21.15.105:0/116829925 --> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa80c000bf0 con 0x7fa80806c4e0 2024-04-06T02:50:46.559 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.574+0000 7fa807fff700 1 -- 172.21.15.105:0/116829925 <== mgr.14221 v2:172.21.15.105:6800/2334550011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fa80c000bf0 con 0x7fa80806c4e0 2024-04-06T02:50:46.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.576+0000 7fa825417700 1 -- 172.21.15.105:0/116829925 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fa80806c4e0 msgr2=0x7fa80806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:46.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.576+0000 7fa825417700 1 --2- 172.21.15.105:0/116829925 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fa80806c4e0 0x7fa80806e9a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa820101c50 tx=0x7fa81000a680 comp rx=0 tx=0).stop 2024-04-06T02:50:46.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.576+0000 7fa825417700 1 -- 172.21.15.105:0/116829925 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa820101df0 msgr2=0x7fa8201968c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:46.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.576+0000 7fa825417700 1 --2- 172.21.15.105:0/116829925 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa820101df0 0x7fa8201968c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa81400bd70 tx=0x7fa81400be50 comp rx=0 tx=0).stop 2024-04-06T02:50:46.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.576+0000 7fa825417700 1 -- 172.21.15.105:0/116829925 shutdown_connections 2024-04-06T02:50:46.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.576+0000 7fa825417700 1 --2- 172.21.15.105:0/116829925 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fa80806c4e0 0x7fa80806e9a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:46.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.576+0000 7fa825417700 1 --2- 172.21.15.105:0/116829925 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa820100bf0 0x7fa820196380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:46.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.576+0000 7fa825417700 1 --2- 172.21.15.105:0/116829925 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa820101df0 0x7fa8201968c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:46.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.576+0000 7fa825417700 1 -- 172.21.15.105:0/116829925 >> 172.21.15.105:0/116829925 conn(0x7fa8200fc190 msgr2=0x7fa820105020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:50:46.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.576+0000 7fa825417700 1 -- 172.21.15.105:0/116829925 shutdown_connections 2024-04-06T02:50:46.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.576+0000 7fa825417700 1 -- 172.21.15.105:0/116829925 wait complete. 2024-04-06T02:50:46.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.684+0000 7f53fd134700 1 -- 172.21.15.105:0/169225035 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53f8103f70 msgr2=0x7f53f8106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:46.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.684+0000 7f53fd134700 1 --2- 172.21.15.105:0/169225035 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53f8103f70 0x7f53f8106360 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f53ec009910 tx=0x7f53ec009c20 comp rx=0 tx=0).stop 2024-04-06T02:50:46.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.684+0000 7f53fd134700 1 -- 172.21.15.105:0/169225035 shutdown_connections 2024-04-06T02:50:46.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.684+0000 7f53fd134700 1 --2- 172.21.15.105:0/169225035 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53f8103f70 0x7f53f8106360 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:46.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.684+0000 7f53fd134700 1 --2- 172.21.15.105:0/169225035 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53f8101640 0x7f53f8103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:46.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.684+0000 7f53fd134700 1 -- 172.21.15.105:0/169225035 >> 172.21.15.105:0/169225035 conn(0x7f53f80fb030 msgr2=0x7f53f80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:50:46.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.684+0000 7f53fd134700 1 -- 172.21.15.105:0/169225035 shutdown_connections 2024-04-06T02:50:46.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.684+0000 7f53fd134700 1 -- 172.21.15.105:0/169225035 wait complete. 2024-04-06T02:50:46.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.685+0000 7f53fd134700 1 Processor -- start 2024-04-06T02:50:46.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.686+0000 7f53fd134700 1 -- start start 2024-04-06T02:50:46.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.686+0000 7f53fd134700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53f8101640 0x7f53f8196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:46.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.686+0000 7f53fd134700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53f8103f70 0x7f53f8196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:46.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.686+0000 7f53fd134700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f8196e90 con 0x7f53f8101640 2024-04-06T02:50:46.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.686+0000 7f53fd134700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f8196fd0 con 0x7f53f8103f70 2024-04-06T02:50:46.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.686+0000 7f53f6d9d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53f8101640 0x7f53f8196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:46.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.686+0000 7f53f6d9d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53f8101640 0x7f53f8196330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40394/0 (socket says 172.21.15.105:40394) 2024-04-06T02:50:46.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.686+0000 7f53f6d9d700 1 -- 172.21.15.105:0/583625051 learned_addr learned my addr 172.21.15.105:0/583625051 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:50:46.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.686+0000 7f53f659c700 1 --2- 172.21.15.105:0/583625051 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53f8103f70 0x7f53f8196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:46.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.686+0000 7f53f659c700 1 -- 172.21.15.105:0/583625051 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53f8101640 msgr2=0x7f53f8196330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:46.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.686+0000 7f53f659c700 1 --2- 172.21.15.105:0/583625051 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53f8101640 0x7f53f8196330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:46.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.686+0000 7f53f659c700 1 -- 172.21.15.105:0/583625051 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ec0095f0 con 0x7f53f8103f70 2024-04-06T02:50:46.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.687+0000 7f53f659c700 1 --2- 172.21.15.105:0/583625051 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53f8103f70 0x7f53f8196870 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f53ec000c00 tx=0x7f53ec00f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:50:46.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.687+0000 7f53dffff700 1 -- 172.21.15.105:0/583625051 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53ec01d070 con 0x7f53f8103f70 2024-04-06T02:50:46.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.687+0000 7f53fd134700 1 -- 172.21.15.105:0/583625051 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f819ba20 con 0x7f53f8103f70 2024-04-06T02:50:46.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.687+0000 7f53fd134700 1 -- 172.21.15.105:0/583625051 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f819bf10 con 0x7f53f8103f70 2024-04-06T02:50:46.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.687+0000 7f53dffff700 1 -- 172.21.15.105:0/583625051 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f53ec005dc0 con 0x7f53f8103f70 2024-04-06T02:50:46.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.687+0000 7f53dffff700 1 -- 172.21.15.105:0/583625051 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53ec017400 con 0x7f53f8103f70 2024-04-06T02:50:46.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.688+0000 7f53fd134700 1 -- 172.21.15.105:0/583625051 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53f8190520 con 0x7f53f8103f70 2024-04-06T02:50:46.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.689+0000 7f53dffff700 1 -- 172.21.15.105:0/583625051 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f53ec00fa40 con 0x7f53f8103f70 2024-04-06T02:50:46.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.689+0000 7f53dffff700 1 --2- 172.21.15.105:0/583625051 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f53e006c4e0 0x7f53e006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:46.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.689+0000 7f53dffff700 1 -- 172.21.15.105:0/583625051 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f53ec08cca0 con 0x7f53f8103f70 2024-04-06T02:50:46.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.689+0000 7f53f6d9d700 1 --2- 172.21.15.105:0/583625051 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f53e006c4e0 0x7f53e006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:46.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.690+0000 7f53f6d9d700 1 --2- 172.21.15.105:0/583625051 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f53e006c4e0 0x7f53e006e9a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f53e8009d20 tx=0x7f53e8009260 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:50:46.679 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.693+0000 7f53dffff700 1 -- 172.21.15.105:0/583625051 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f53ec05aed0 con 0x7f53f8103f70 2024-04-06T02:50:46.911 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.926+0000 7f53fd134700 1 -- 172.21.15.105:0/583625051 --> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f53f80611d0 con 0x7f53e006c4e0 2024-04-06T02:50:46.922 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.937+0000 7f53dffff700 1 -- 172.21.15.105:0/583625051 <== mgr.14221 v2:172.21.15.105:6800/2334550011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3528 (secure 0 0 0) 0x7f53f80611d0 con 0x7f53e006c4e0 2024-04-06T02:50:46.922 INFO:teuthology.orchestra.run.smithi105.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-06T02:50:46.922 INFO:teuthology.orchestra.run.smithi105.stdout:alertmanager.smithi105 smithi105 *:9093,9094 running (6m) 94s ago 7m 25.4M - 0.25.0 c8568f914cd2 5b927ee87a96 2024-04-06T02:50:46.923 INFO:teuthology.orchestra.run.smithi105.stdout:ceph-exporter.smithi105 smithi105 running (7m) 94s ago 7m 7453k - 18.2.1 5be31c24972a 448b72643fea 2024-04-06T02:50:46.923 INFO:teuthology.orchestra.run.smithi105.stdout:ceph-exporter.smithi184 smithi184 running (6m) 99s ago 6m 7377k - 18.2.1 5be31c24972a 52fd408df95f 2024-04-06T02:50:46.923 INFO:teuthology.orchestra.run.smithi105.stdout:crash.smithi105 smithi105 running (7m) 94s ago 7m 6660k - 18.2.1 5be31c24972a 8e21a7fab0b2 2024-04-06T02:50:46.923 INFO:teuthology.orchestra.run.smithi105.stdout:crash.smithi184 smithi184 running (6m) 99s ago 6m 6660k - 18.2.1 5be31c24972a a0c065bc6908 2024-04-06T02:50:46.923 INFO:teuthology.orchestra.run.smithi105.stdout:grafana.smithi105 smithi105 *:3000 running (5m) 94s ago 7m 81.1M - 9.4.7 954c08fa6188 413a7bf83436 2024-04-06T02:50:46.923 INFO:teuthology.orchestra.run.smithi105.stdout:mds.cephfs.smithi105.mrivth smithi105 running (2m) 94s ago 119s 11.3M - 18.2.1 5be31c24972a c682a8e71935 2024-04-06T02:50:46.923 INFO:teuthology.orchestra.run.smithi105.stdout:mds.cephfs.smithi105.oxxcbq smithi105 running (113s) 94s ago 113s 18.2M - 18.2.1 5be31c24972a 9ca38954f88a 2024-04-06T02:50:46.923 INFO:teuthology.orchestra.run.smithi105.stdout:mds.cephfs.smithi184.htgrjn smithi184 running (110s) 99s ago 110s 15.3M - 18.2.1 5be31c24972a d6049e6eb483 2024-04-06T02:50:46.923 INFO:teuthology.orchestra.run.smithi105.stdout:mds.cephfs.smithi184.oochxp smithi184 running (117s) 99s ago 117s 16.0M - 18.2.1 5be31c24972a fd16feedab03 2024-04-06T02:50:46.923 INFO:teuthology.orchestra.run.smithi105.stdout:mgr.smithi105.vqqcpx smithi105 *:9283,8765,8443 running (9m) 94s ago 9m 489M - 18.2.1 5be31c24972a 07c0a110a1b8 2024-04-06T02:50:46.923 INFO:teuthology.orchestra.run.smithi105.stdout:mgr.smithi184.fzhhql smithi184 *:8443,9283,8765 running (6m) 99s ago 6m 443M - 18.2.1 5be31c24972a a09137f55e51 2024-04-06T02:50:46.923 INFO:teuthology.orchestra.run.smithi105.stdout:mon.smithi105 smithi105 running (9m) 94s ago 9m 46.7M 2048M 18.2.1 5be31c24972a d27bf0498159 2024-04-06T02:50:46.923 INFO:teuthology.orchestra.run.smithi105.stdout:mon.smithi184 smithi184 running (6m) 99s ago 6m 38.9M 2048M 18.2.1 5be31c24972a 9054b877f58f 2024-04-06T02:50:46.923 INFO:teuthology.orchestra.run.smithi105.stdout:node-exporter.smithi105 smithi105 *:9100 running (7m) 94s ago 7m 15.9M - 1.5.0 0da6a335fe13 543e8799de93 2024-04-06T02:50:46.924 INFO:teuthology.orchestra.run.smithi105.stdout:node-exporter.smithi184 smithi184 *:9100 running (6m) 99s ago 6m 15.7M - 1.5.0 0da6a335fe13 7688716a6f4a 2024-04-06T02:50:46.924 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0 smithi105 running (5m) 94s ago 5m 45.9M 3946M 18.2.1 5be31c24972a 339c3f865ef9 2024-04-06T02:50:46.924 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1 smithi105 running (5m) 94s ago 5m 48.5M 3946M 18.2.1 5be31c24972a eb4cf9ea6b1b 2024-04-06T02:50:46.924 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2 smithi105 running (4m) 94s ago 4m 45.8M 3946M 18.2.1 5be31c24972a 0392f09bdb6b 2024-04-06T02:50:46.924 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3 smithi184 running (4m) 99s ago 4m 47.2M 4970M 18.2.1 5be31c24972a e833150f2893 2024-04-06T02:50:46.924 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4 smithi184 running (3m) 99s ago 3m 44.2M 4970M 18.2.1 5be31c24972a a13051e8d0fb 2024-04-06T02:50:46.924 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5 smithi184 running (3m) 99s ago 3m 46.9M 4970M 18.2.1 5be31c24972a b7b1ce8932de 2024-04-06T02:50:46.924 INFO:teuthology.orchestra.run.smithi105.stdout:prometheus.smithi105 smithi105 *:9095 running (5m) 94s ago 7m 52.7M - 2.43.0 a07b618ecd1d ee5967404456 2024-04-06T02:50:46.925 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.938+0000 7f53fd134700 1 -- 172.21.15.105:0/583625051 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f53e006c4e0 msgr2=0x7f53e006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:46.925 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.938+0000 7f53fd134700 1 --2- 172.21.15.105:0/583625051 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f53e006c4e0 0x7f53e006e9a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f53e8009d20 tx=0x7f53e8009260 comp rx=0 tx=0).stop 2024-04-06T02:50:46.925 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.938+0000 7f53fd134700 1 -- 172.21.15.105:0/583625051 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53f8103f70 msgr2=0x7f53f8196870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:46.925 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.939+0000 7f53fd134700 1 --2- 172.21.15.105:0/583625051 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53f8103f70 0x7f53f8196870 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f53ec000c00 tx=0x7f53ec00f690 comp rx=0 tx=0).stop 2024-04-06T02:50:46.925 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.939+0000 7f53fd134700 1 -- 172.21.15.105:0/583625051 shutdown_connections 2024-04-06T02:50:46.925 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.939+0000 7f53fd134700 1 --2- 172.21.15.105:0/583625051 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f53e006c4e0 0x7f53e006e9a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:46.925 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.939+0000 7f53fd134700 1 --2- 172.21.15.105:0/583625051 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53f8101640 0x7f53f8196330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:46.925 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.939+0000 7f53fd134700 1 --2- 172.21.15.105:0/583625051 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f53f8103f70 0x7f53f8196870 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:46.925 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.939+0000 7f53fd134700 1 -- 172.21.15.105:0/583625051 >> 172.21.15.105:0/583625051 conn(0x7f53f80fb030 msgr2=0x7f53f80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:50:46.925 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.939+0000 7f53fd134700 1 -- 172.21.15.105:0/583625051 shutdown_connections 2024-04-06T02:50:46.925 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:46.939+0000 7f53fd134700 1 -- 172.21.15.105:0/583625051 wait complete. 2024-04-06T02:50:47.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.045+0000 7f8dd779a700 1 -- 172.21.15.105:0/4187998050 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8dd00ffab0 msgr2=0x7f8dd00ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:47.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.045+0000 7f8dd779a700 1 --2- 172.21.15.105:0/4187998050 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8dd00ffab0 0x7f8dd00ffed0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc009910 tx=0x7f8dcc009c20 comp rx=0 tx=0).stop 2024-04-06T02:50:47.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.045+0000 7f8dd779a700 1 -- 172.21.15.105:0/4187998050 shutdown_connections 2024-04-06T02:50:47.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.045+0000 7f8dd779a700 1 --2- 172.21.15.105:0/4187998050 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8dd0100cb0 0x7f8dd0101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:47.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.045+0000 7f8dd779a700 1 --2- 172.21.15.105:0/4187998050 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8dd00ffab0 0x7f8dd00ffed0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:47.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.045+0000 7f8dd779a700 1 -- 172.21.15.105:0/4187998050 >> 172.21.15.105:0/4187998050 conn(0x7f8dd00fb030 msgr2=0x7f8dd00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:50:47.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.045+0000 7f8dd779a700 1 -- 172.21.15.105:0/4187998050 shutdown_connections 2024-04-06T02:50:47.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.045+0000 7f8dd779a700 1 -- 172.21.15.105:0/4187998050 wait complete. 2024-04-06T02:50:47.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.046+0000 7f8dd779a700 1 Processor -- start 2024-04-06T02:50:47.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.046+0000 7f8dd779a700 1 -- start start 2024-04-06T02:50:47.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.046+0000 7f8dd779a700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8dd00ffab0 0x7f8dd0068f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:47.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.046+0000 7f8dd779a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8dd0100cb0 0x7f8dd019d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:47.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.046+0000 7f8dd779a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd0196f10 con 0x7f8dd0100cb0 2024-04-06T02:50:47.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.046+0000 7f8dd779a700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd019d9e0 con 0x7f8dd00ffab0 2024-04-06T02:50:47.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.046+0000 7f8dd4d35700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8dd0100cb0 0x7f8dd019d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:47.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.046+0000 7f8dd5536700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8dd00ffab0 0x7f8dd0068f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:47.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.046+0000 7f8dd4d35700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8dd0100cb0 0x7f8dd019d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40400/0 (socket says 172.21.15.105:40400) 2024-04-06T02:50:47.034 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.046+0000 7f8dd4d35700 1 -- 172.21.15.105:0/3507235127 learned_addr learned my addr 172.21.15.105:0/3507235127 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:50:47.034 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.047+0000 7f8dd4d35700 1 -- 172.21.15.105:0/3507235127 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8dd00ffab0 msgr2=0x7f8dd0068f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:47.034 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.047+0000 7f8dd4d35700 1 --2- 172.21.15.105:0/3507235127 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8dd00ffab0 0x7f8dd0068f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:47.034 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.047+0000 7f8dd4d35700 1 -- 172.21.15.105:0/3507235127 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dcc0095f0 con 0x7f8dd0100cb0 2024-04-06T02:50:47.034 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.047+0000 7f8dd5536700 1 --2- 172.21.15.105:0/3507235127 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8dd00ffab0 0x7f8dd0068f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T02:50:47.034 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.047+0000 7f8dd4d35700 1 --2- 172.21.15.105:0/3507235127 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8dd0100cb0 0x7f8dd019d4a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8dc000e950 tx=0x7f8dc000ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:50:47.034 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.047+0000 7f8dc67fc700 1 -- 172.21.15.105:0/3507235127 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8dc000cba0 con 0x7f8dd0100cb0 2024-04-06T02:50:47.034 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.047+0000 7f8dd779a700 1 -- 172.21.15.105:0/3507235127 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd019dc90 con 0x7f8dd0100cb0 2024-04-06T02:50:47.034 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.047+0000 7f8dc67fc700 1 -- 172.21.15.105:0/3507235127 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8dc000eea0 con 0x7f8dd0100cb0 2024-04-06T02:50:47.034 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.047+0000 7f8dc67fc700 1 -- 172.21.15.105:0/3507235127 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8dc0018650 con 0x7f8dd0100cb0 2024-04-06T02:50:47.034 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.047+0000 7f8dd779a700 1 -- 172.21.15.105:0/3507235127 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd019e1e0 con 0x7f8dd0100cb0 2024-04-06T02:50:47.035 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.048+0000 7f8dd779a700 1 -- 172.21.15.105:0/3507235127 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd0108610 con 0x7f8dd0100cb0 2024-04-06T02:50:47.035 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.049+0000 7f8dc67fc700 1 -- 172.21.15.105:0/3507235127 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f8dc000cd00 con 0x7f8dd0100cb0 2024-04-06T02:50:47.035 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.050+0000 7f8dc67fc700 1 --2- 172.21.15.105:0/3507235127 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f8dbc06c4e0 0x7f8dbc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:47.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.050+0000 7f8dc67fc700 1 -- 172.21.15.105:0/3507235127 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f8dc0014070 con 0x7f8dd0100cb0 2024-04-06T02:50:47.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.050+0000 7f8dd5536700 1 --2- 172.21.15.105:0/3507235127 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f8dbc06c4e0 0x7f8dbc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:47.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.050+0000 7f8dd5536700 1 --2- 172.21.15.105:0/3507235127 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f8dbc06c4e0 0x7f8dbc06e9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc00b6f0 tx=0x7f8dcc005940 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:50:47.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.058+0000 7f8dc67fc700 1 -- 172.21.15.105:0/3507235127 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8dc00520e0 con 0x7f8dd0100cb0 2024-04-06T02:50:47.138 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:46 smithi105 ceph-mon[29228]: Upgrade: Updating mgr.smithi184.fzhhql 2024-04-06T02:50:47.138 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:46 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:50:47.138 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:46 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi184.fzhhql", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:50:47.138 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:46 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:50:47.138 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:46 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:50:47.139 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:46 smithi105 ceph-mon[29228]: Deploying daemon mgr.smithi184.fzhhql on smithi184 2024-04-06T02:50:47.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:46 smithi184 ceph-mon[37439]: Upgrade: Updating mgr.smithi184.fzhhql 2024-04-06T02:50:47.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:46 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' 2024-04-06T02:50:47.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:46 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi184.fzhhql", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T02:50:47.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:46 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T02:50:47.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:46 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T02:50:47.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:46 smithi184 ceph-mon[37439]: Deploying daemon mgr.smithi184.fzhhql on smithi184 2024-04-06T02:50:47.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.377+0000 7f8dd779a700 1 -- 172.21.15.105:0/3507235127 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8dd0108800 con 0x7f8dd0100cb0 2024-04-06T02:50:47.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.378+0000 7f8dc67fc700 1 -- 172.21.15.105:0/3507235127 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7f8dc0059d10 con 0x7f8dd0100cb0 2024-04-06T02:50:47.364 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-06T02:50:47.364 INFO:teuthology.orchestra.run.smithi105.stdout: "mon": { 2024-04-06T02:50:47.364 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-06T02:50:47.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-06T02:50:47.364 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr": { 2024-04-06T02:50:47.364 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-06T02:50:47.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-06T02:50:47.364 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": { 2024-04-06T02:50:47.364 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-06T02:50:47.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-06T02:50:47.364 INFO:teuthology.orchestra.run.smithi105.stdout: "mds": { 2024-04-06T02:50:47.364 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-06T02:50:47.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-06T02:50:47.364 INFO:teuthology.orchestra.run.smithi105.stdout: "overall": { 2024-04-06T02:50:47.364 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 14 2024-04-06T02:50:47.365 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-06T02:50:47.365 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-06T02:50:47.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.380+0000 7f8dd779a700 1 -- 172.21.15.105:0/3507235127 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f8dbc06c4e0 msgr2=0x7f8dbc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:47.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.380+0000 7f8dd779a700 1 --2- 172.21.15.105:0/3507235127 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f8dbc06c4e0 0x7f8dbc06e9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc00b6f0 tx=0x7f8dcc005940 comp rx=0 tx=0).stop 2024-04-06T02:50:47.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.380+0000 7f8dd779a700 1 -- 172.21.15.105:0/3507235127 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8dd0100cb0 msgr2=0x7f8dd019d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:47.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.380+0000 7f8dd779a700 1 --2- 172.21.15.105:0/3507235127 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8dd0100cb0 0x7f8dd019d4a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8dc000e950 tx=0x7f8dc000ec60 comp rx=0 tx=0).stop 2024-04-06T02:50:47.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.380+0000 7f8dd779a700 1 -- 172.21.15.105:0/3507235127 shutdown_connections 2024-04-06T02:50:47.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.380+0000 7f8dd779a700 1 --2- 172.21.15.105:0/3507235127 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7f8dbc06c4e0 0x7f8dbc06e9a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:47.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.380+0000 7f8dd779a700 1 --2- 172.21.15.105:0/3507235127 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8dd00ffab0 0x7f8dd0068f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:47.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.380+0000 7f8dd779a700 1 --2- 172.21.15.105:0/3507235127 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8dd0100cb0 0x7f8dd019d4a0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:47.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.380+0000 7f8dd779a700 1 -- 172.21.15.105:0/3507235127 >> 172.21.15.105:0/3507235127 conn(0x7f8dd00fb030 msgr2=0x7f8dd0103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:50:47.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.380+0000 7f8dd779a700 1 -- 172.21.15.105:0/3507235127 shutdown_connections 2024-04-06T02:50:47.367 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.380+0000 7f8dd779a700 1 -- 172.21.15.105:0/3507235127 wait complete. 2024-04-06T02:50:47.518 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.530+0000 7fd1f0dd1700 1 -- 172.21.15.105:0/3086168091 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1ec101df0 msgr2=0x7fd1ec102250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:47.518 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.530+0000 7fd1f0dd1700 1 --2- 172.21.15.105:0/3086168091 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1ec101df0 0x7fd1ec102250 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0009910 tx=0x7fd1e0009c20 comp rx=0 tx=0).stop 2024-04-06T02:50:47.518 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.530+0000 7fd1f0dd1700 1 -- 172.21.15.105:0/3086168091 shutdown_connections 2024-04-06T02:50:47.518 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.530+0000 7fd1f0dd1700 1 --2- 172.21.15.105:0/3086168091 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1ec101df0 0x7fd1ec102250 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:47.518 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.530+0000 7fd1f0dd1700 1 --2- 172.21.15.105:0/3086168091 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1ec100bf0 0x7fd1ec101010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:47.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.530+0000 7fd1f0dd1700 1 -- 172.21.15.105:0/3086168091 >> 172.21.15.105:0/3086168091 conn(0x7fd1ec0fc190 msgr2=0x7fd1ec0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:50:47.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.530+0000 7fd1f0dd1700 1 -- 172.21.15.105:0/3086168091 shutdown_connections 2024-04-06T02:50:47.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.530+0000 7fd1f0dd1700 1 -- 172.21.15.105:0/3086168091 wait complete. 2024-04-06T02:50:47.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.531+0000 7fd1f0dd1700 1 Processor -- start 2024-04-06T02:50:47.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.531+0000 7fd1f0dd1700 1 -- start start 2024-04-06T02:50:47.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.531+0000 7fd1f0dd1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1ec100bf0 0x7fd1ec196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:47.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.531+0000 7fd1f0dd1700 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1ec101df0 0x7fd1ec1968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:47.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.531+0000 7fd1f0dd1700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec196ee0 con 0x7fd1ec100bf0 2024-04-06T02:50:47.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.531+0000 7fd1f0dd1700 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec197020 con 0x7fd1ec101df0 2024-04-06T02:50:47.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.531+0000 7fd1ea59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1ec100bf0 0x7fd1ec196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:47.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.532+0000 7fd1ea59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1ec100bf0 0x7fd1ec196380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40424/0 (socket says 172.21.15.105:40424) 2024-04-06T02:50:47.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.532+0000 7fd1ea59c700 1 -- 172.21.15.105:0/649157263 learned_addr learned my addr 172.21.15.105:0/649157263 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-06T02:50:47.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.532+0000 7fd1e9d9b700 1 --2- 172.21.15.105:0/649157263 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1ec101df0 0x7fd1ec1968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:47.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.532+0000 7fd1e9d9b700 1 -- 172.21.15.105:0/649157263 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1ec100bf0 msgr2=0x7fd1ec196380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:47.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.532+0000 7fd1e9d9b700 1 --2- 172.21.15.105:0/649157263 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1ec100bf0 0x7fd1ec196380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:47.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.532+0000 7fd1e9d9b700 1 -- 172.21.15.105:0/649157263 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e00095f0 con 0x7fd1ec101df0 2024-04-06T02:50:47.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.532+0000 7fd1ea59c700 1 --2- 172.21.15.105:0/649157263 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1ec100bf0 0x7fd1ec196380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-06T02:50:47.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.532+0000 7fd1e9d9b700 1 --2- 172.21.15.105:0/649157263 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1ec101df0 0x7fd1ec1968c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0009910 tx=0x7fd1e000bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:50:47.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.532+0000 7fd1db7fe700 1 -- 172.21.15.105:0/649157263 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1e001d070 con 0x7fd1ec101df0 2024-04-06T02:50:47.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.532+0000 7fd1f0dd1700 1 -- 172.21.15.105:0/649157263 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1ec19ba70 con 0x7fd1ec101df0 2024-04-06T02:50:47.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.532+0000 7fd1db7fe700 1 -- 172.21.15.105:0/649157263 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd1e000f460 con 0x7fd1ec101df0 2024-04-06T02:50:47.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.532+0000 7fd1db7fe700 1 -- 172.21.15.105:0/649157263 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1e0021620 con 0x7fd1ec101df0 2024-04-06T02:50:47.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.533+0000 7fd1f0dd1700 1 -- 172.21.15.105:0/649157263 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1ec19bf60 con 0x7fd1ec101df0 2024-04-06T02:50:47.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.534+0000 7fd1f0dd1700 1 -- 172.21.15.105:0/649157263 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1ec109750 con 0x7fd1ec101df0 2024-04-06T02:50:47.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.534+0000 7fd1db7fe700 1 -- 172.21.15.105:0/649157263 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fd1e0003a60 con 0x7fd1ec101df0 2024-04-06T02:50:47.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.534+0000 7fd1db7fe700 1 --2- 172.21.15.105:0/649157263 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fd1d406c4e0 0x7fd1d406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-06T02:50:47.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.534+0000 7fd1db7fe700 1 -- 172.21.15.105:0/649157263 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fd1e008ccd0 con 0x7fd1ec101df0 2024-04-06T02:50:47.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.535+0000 7fd1ea59c700 1 --2- 172.21.15.105:0/649157263 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fd1d406c4e0 0x7fd1d406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-06T02:50:47.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.535+0000 7fd1ea59c700 1 --2- 172.21.15.105:0/649157263 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fd1d406c4e0 0x7fd1d406e9a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc0105c0 tx=0x7fd1dc00b4a0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-06T02:50:47.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.543+0000 7fd1db7fe700 1 -- 172.21.15.105:0/649157263 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd1e005b000 con 0x7fd1ec101df0 2024-04-06T02:50:47.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.902+0000 7fd1f0dd1700 1 -- 172.21.15.105:0/649157263 --> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1ec106740 con 0x7fd1d406c4e0 2024-04-06T02:50:47.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.904+0000 7fd1db7fe700 1 -- 172.21.15.105:0/649157263 <== mgr.14221 v2:172.21.15.105:6800/2334550011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fd1ec106740 con 0x7fd1d406c4e0 2024-04-06T02:50:47.889 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-06T02:50:47.889 INFO:teuthology.orchestra.run.smithi105.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T02:50:47.890 INFO:teuthology.orchestra.run.smithi105.stdout: "in_progress": true, 2024-04-06T02:50:47.890 INFO:teuthology.orchestra.run.smithi105.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-06T02:50:47.890 INFO:teuthology.orchestra.run.smithi105.stdout: "services_complete": [], 2024-04-06T02:50:47.890 INFO:teuthology.orchestra.run.smithi105.stdout: "progress": "0/2 daemons upgraded", 2024-04-06T02:50:47.890 INFO:teuthology.orchestra.run.smithi105.stdout: "message": "Currently upgrading mgr daemons", 2024-04-06T02:50:47.890 INFO:teuthology.orchestra.run.smithi105.stdout: "is_paused": false 2024-04-06T02:50:47.890 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-06T02:50:47.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.907+0000 7fd1f0dd1700 1 -- 172.21.15.105:0/649157263 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fd1d406c4e0 msgr2=0x7fd1d406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:47.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.907+0000 7fd1f0dd1700 1 --2- 172.21.15.105:0/649157263 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fd1d406c4e0 0x7fd1d406e9a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc0105c0 tx=0x7fd1dc00b4a0 comp rx=0 tx=0).stop 2024-04-06T02:50:47.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.907+0000 7fd1f0dd1700 1 -- 172.21.15.105:0/649157263 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1ec101df0 msgr2=0x7fd1ec1968c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-06T02:50:47.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.907+0000 7fd1f0dd1700 1 --2- 172.21.15.105:0/649157263 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1ec101df0 0x7fd1ec1968c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0009910 tx=0x7fd1e000bd70 comp rx=0 tx=0).stop 2024-04-06T02:50:47.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.907+0000 7fd1f0dd1700 1 -- 172.21.15.105:0/649157263 shutdown_connections 2024-04-06T02:50:47.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.907+0000 7fd1f0dd1700 1 --2- 172.21.15.105:0/649157263 >> [v2:172.21.15.105:6800/2334550011,v1:172.21.15.105:6801/2334550011] conn(0x7fd1d406c4e0 0x7fd1d406e9a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:47.895 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.907+0000 7fd1f0dd1700 1 --2- 172.21.15.105:0/649157263 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1ec100bf0 0x7fd1ec196380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:47.895 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.907+0000 7fd1f0dd1700 1 --2- 172.21.15.105:0/649157263 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1ec101df0 0x7fd1ec1968c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-06T02:50:47.895 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.907+0000 7fd1f0dd1700 1 -- 172.21.15.105:0/649157263 >> 172.21.15.105:0/649157263 conn(0x7fd1ec0fc190 msgr2=0x7fd1ec105020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-06T02:50:47.895 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.907+0000 7fd1f0dd1700 1 -- 172.21.15.105:0/649157263 shutdown_connections 2024-04-06T02:50:47.895 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:50:47.907+0000 7fd1f0dd1700 1 -- 172.21.15.105:0/649157263 wait complete. 2024-04-06T02:50:48.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:47 smithi105 ceph-mon[29228]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:50:48.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:47 smithi105 ceph-mon[29228]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:50:48.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:47 smithi105 ceph-mon[29228]: pgmap v241: 65 pgs: 65 active+clean; 68 MiB data, 425 MiB used, 536 GiB / 536 GiB avail; 5.9 MiB/s wr, 466 op/s 2024-04-06T02:50:48.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:47 smithi105 ceph-mon[29228]: from='client.24333 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:50:48.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:47 smithi105 ceph-mon[29228]: from='client.? 172.21.15.105:0/3507235127' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:50:48.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:47 smithi184 ceph-mon[37439]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:50:48.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:47 smithi184 ceph-mon[37439]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:50:48.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:47 smithi184 ceph-mon[37439]: pgmap v241: 65 pgs: 65 active+clean; 68 MiB data, 425 MiB used, 536 GiB / 536 GiB avail; 5.9 MiB/s wr, 466 op/s 2024-04-06T02:50:48.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:47 smithi184 ceph-mon[37439]: from='client.24333 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:50:48.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:47 smithi184 ceph-mon[37439]: from='client.? 172.21.15.105:0/3507235127' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-06T02:50:49.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:48 smithi105 ceph-mon[29228]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:50:49.230 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:48 smithi184 ceph-mon[37439]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T02:50:50.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:49 smithi105 ceph-mon[29228]: pgmap v242: 65 pgs: 65 active+clean; 74 MiB data, 434 MiB used, 536 GiB / 536 GiB avail; 6.2 MiB/s wr, 466 op/s 2024-04-06T02:50:50.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:49 smithi184 ceph-mon[37439]: pgmap v242: 65 pgs: 65 active+clean; 74 MiB data, 434 MiB used, 536 GiB / 536 GiB avail; 6.2 MiB/s wr, 466 op/s 2024-04-06T02:50:52.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:51 smithi105 ceph-mon[29228]: pgmap v243: 65 pgs: 65 active+clean; 92 MiB data, 466 MiB used, 536 GiB / 536 GiB avail; 7.5 MiB/s wr, 568 op/s 2024-04-06T02:50:52.231 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:51 smithi184 ceph-mon[37439]: pgmap v243: 65 pgs: 65 active+clean; 92 MiB data, 466 MiB used, 536 GiB / 536 GiB avail; 7.5 MiB/s wr, 568 op/s 2024-04-06T02:50:53.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:53 smithi105 ceph-mon[29228]: pgmap v244: 65 pgs: 65 active+clean; 117 MiB data, 561 MiB used, 536 GiB / 536 GiB avail; 7.6 MiB/s wr, 657 op/s 2024-04-06T02:50:53.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:53 smithi184 ceph-mon[37439]: pgmap v244: 65 pgs: 65 active+clean; 117 MiB data, 561 MiB used, 536 GiB / 536 GiB avail; 7.6 MiB/s wr, 657 op/s 2024-04-06T02:50:54.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:54 smithi184 ceph-mon[37439]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:50:55.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:54 smithi105 ceph-mon[29228]: from='mgr.14221 172.21.15.105:0/3641715811' entity='mgr.smithi105.vqqcpx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T02:50:55.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:55 smithi184 ceph-mon[37439]: pgmap v245: 65 pgs: 65 active+clean; 117 MiB data, 561 MiB used, 536 GiB / 536 GiB avail; 5.2 MiB/s wr, 422 op/s 2024-04-06T02:50:56.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:55 smithi105 ceph-mon[29228]: pgmap v245: 65 pgs: 65 active+clean; 117 MiB data, 561 MiB used, 536 GiB / 536 GiB avail; 5.2 MiB/s wr, 422 op/s 2024-04-06T02:50:57.024 INFO:tasks.workunit.client.0.smithi105.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-06T02:50:57.025 INFO:tasks.workunit.client.0.smithi105.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-06T02:50:57.025 INFO:tasks.workunit.client.0.smithi105.stderr:+ make 2024-04-06T02:50:57.443 INFO:tasks.workunit.client.1.smithi184.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-06T02:50:57.454 INFO:tasks.workunit.client.1.smithi184.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-06T02:50:57.454 INFO:tasks.workunit.client.1.smithi184.stderr:+ make 2024-04-06T02:50:57.483 INFO:tasks.workunit.client.0.smithi105.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-06T02:50:57.559 INFO:tasks.workunit.client.1.smithi184.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-06T02:50:57.981 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:Apr 06 02:50:57 smithi184 ceph-mon[37439]: pgmap v246: 65 pgs: 65 active+clean; 169 MiB data, 793 MiB used, 536 GiB / 536 GiB avail; 9.7 MiB/s wr, 926 op/s 2024-04-06T02:50:58.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:57 smithi105 ceph-mon[29228]: pgmap v246: 65 pgs: 65 active+clean; 169 MiB data, 793 MiB used, 536 GiB / 536 GiB avail; 9.7 MiB/s wr, 926 op/s 2024-04-06T02:51:00.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:50:59 smithi105 ceph-mon[29228]: pgmap v247: 65 pgs: 65 active+clean; 182 MiB data, 866 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s wr, 995 op/s 2024-04-06T02:51:21.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:20.942+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:21.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:20.942+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:21.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:20.942+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:22.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:21.919+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:22.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:21.919+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:22.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:21.919+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:23.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:22.886+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:23.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:22.886+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:23.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:22.886+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:23.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:23.026+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:23.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:23.026+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:23.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:23.026+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:24.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:23.988+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:24.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:23.988+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:24.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:23.988+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:24.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:23.931+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:24.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:23.931+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:24.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:23.931+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:24.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:24.226+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:24.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:24.226+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:24.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:24.226+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:25.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:25.029+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:25.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:25.029+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:25.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:25.029+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:25.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:24.892+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:25.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:24.892+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:25.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:24.892+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:25.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:25.242+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:25.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:25.242+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:25.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:25.242+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:26.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:26.027+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:26.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:26.027+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:26.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:26.027+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:26.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:25.894+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:26.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:25.894+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:26.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:25.894+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:26.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:26.254+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:26.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:26.254+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:26.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:26.254+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:27.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:27.073+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:27.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:27.073+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:27.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:27.073+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:27.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:26.869+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:27.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:26.869+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:27.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:26.869+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:27.503 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:27.263+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:27.504 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:27.263+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:27.504 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:27.263+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:28.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:28.119+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:28.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:28.119+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:28.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:28.119+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:28.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:27.906+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:28.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:27.906+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:28.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:27.906+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:28.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:28.309+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:28.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:28.309+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:28.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:28.309+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:29.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:29.070+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:29.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:29.070+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:29.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:29.070+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:29.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:28.907+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:29.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:28.907+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:29.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:28.907+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:29.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:29.339+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:29.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:29.339+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:29.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:29.339+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:30.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:30.069+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:30.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:30.069+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:30.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:30.069+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:30.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:29.930+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:30.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:29.930+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:30.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:29.930+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:30.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:30.350+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:30.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:30.350+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:30.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:30.350+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:31.314 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:31.054+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:31.314 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:31.054+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:31.314 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:31.054+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:31.314 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:30.961+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:31.315 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:30.961+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:31.315 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:30.961+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:31.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:31.328+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:31.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:31.328+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:31.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:31.328+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:32.324 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:32.038+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:32.324 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:32.038+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:32.324 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:32.038+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:32.324 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:31.970+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:32.324 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:31.970+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:32.325 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:31.970+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:32.628 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:32.339+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:32.628 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:32.339+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:32.629 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:32.339+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:33.333 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:32.989+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:33.333 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:32.990+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:33.333 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:32.990+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:33.333 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:33.018+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:33.333 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:33.018+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:33.333 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:33.018+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:33.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:33.347+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:33.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:33.347+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:33.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:33.347+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:34.356 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:34.015+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:34.356 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:34.015+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:34.356 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:34.015+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:34.356 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:34.052+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:34.356 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:34.052+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:34.356 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:34.052+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:34.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:34.370+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:34.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:34.370+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:34.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:34.370+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:35.398 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:35.020+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:35.398 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:35.020+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:35.398 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:35.020+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:35.399 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:35.059+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:35.399 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:35.059+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:35.399 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:35.059+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:35.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:35.413+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:35.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:35.413+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:35.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:35.413+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:36.379 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:36.032+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:36.379 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:36.032+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:36.380 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:36.032+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:36.380 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:36.015+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:36.380 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:36.015+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:36.380 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:36.015+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:36.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:36.394+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:36.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:36.394+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:36.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:36.394+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:37.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:37.054+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:37.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:37.054+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:37.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:37.054+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:37.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:37.044+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:37.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:37.044+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:37.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:37.044+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:37.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:37.439+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:37.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:37.439+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:37.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:37.439+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:38.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:38.089+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:38.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:38.089+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:38.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:38.089+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:38.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:38.068+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:38.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:38.068+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:38.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:38.068+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:38.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:38.465+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:38.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:38.465+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:38.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:38.465+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:39.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:39.057+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:39.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:39.057+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:39.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:39.057+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:39.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:39.026+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:39.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:39.026+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:39.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:39.026+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:39.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:39.483+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:39.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:39.483+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:39.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:39.483+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:40.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:40.056+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:40.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:40.056+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:40.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:40.056+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:40.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:40.053+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:40.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:40.053+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:40.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:40.053+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:40.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:40.490+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:40.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:40.490+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:40.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:40.490+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:41.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:41.062+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:41.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:41.062+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:41.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:41.062+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:41.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:41.014+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:41.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:41.014+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:41.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:41.014+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:41.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:41.518+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:41.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:41.518+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:41.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:41.518+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:42.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:42.060+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:42.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:42.060+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:42.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:42.060+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:42.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:42.003+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:42.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:42.003+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:42.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:42.003+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:42.762 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:42.483+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:42.762 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:42.483+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:42.762 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:42.483+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:43.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:43.011+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:43.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:43.011+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:43.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:43.011+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:43.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:42.990+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:43.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:42.990+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:43.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:42.990+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:43.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:43.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:43.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:43.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:43.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:43.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:44.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:43.975+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:44.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:43.975+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:44.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:43.975+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:44.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:44.020+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:44.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:44.020+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:44.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:44.020+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:44.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:44.535+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:44.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:44.535+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:44.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:44.535+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:45.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:45.001+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:45.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:45.001+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:45.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:45.001+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:45.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:45.024+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:45.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:45.024+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:45.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:45.024+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:45.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:45.528+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:45.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:45.528+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:45.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:45.528+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:46.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:46.018+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:46.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:46.018+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:46.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:46.018+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:46.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:45.983+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:46.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:45.983+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:46.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:45.983+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:46.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:46.499+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:46.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:46.499+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:46.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:46.499+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:47.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:47.007+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:47.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:47.007+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:47.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:47.007+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:47.427 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:46.959+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:47.427 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:46.959+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:47.427 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:46.959+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:47.813 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:47.529+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:47.813 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:47.529+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:47.813 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:47.529+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:48.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:48.043+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:48.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:48.043+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:48.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:48.043+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:48.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:47.933+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:48.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:47.933+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:48.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:47.933+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:48.884 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:48.540+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:48.884 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:48.540+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:48.884 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:48.540+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:49.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:49.009+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:49.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:49.009+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:49.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:49.009+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:49.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:48.898+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:49.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:48.898+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:49.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:48.898+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:49.895 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:49.550+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:49.895 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:49.550+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:49.895 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:49.550+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:50.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:50.001+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:50.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:50.001+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:50.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:50.001+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:50.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:49.909+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:50.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:49.909+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:50.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:49.909+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:50.850 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:50.599+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:50.850 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:50.599+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:50.850 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:50.599+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:51.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:50.993+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:51.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:50.993+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:51.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:50.993+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:51.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:50.864+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:51.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:50.864+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:51.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:50.864+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:51.864 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:51.597+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:51.864 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:51.597+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:51.864 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:51.597+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:52.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:52.012+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:52.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:52.013+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:52.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:52.013+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:52.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:51.878+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:52.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:51.878+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:52.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:51.878+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:52.863 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:52.565+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:52.863 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:52.565+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:52.863 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:52.565+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:53.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:53.035+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:53.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:53.035+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:53.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:53.035+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:53.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:52.879+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:53.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:52.879+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:53.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:52.879+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:53.902 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:53.581+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:53.902 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:53.581+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:53.902 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:53.581+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:54.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:54.033+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:54.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:54.033+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:54.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:54.033+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:54.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:53.916+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:54.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:53.916+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:54.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:53.916+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:54.887 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:54.570+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:54.888 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:54.570+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:54.888 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:54.570+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:55.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:55.081+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:55.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:55.081+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:55.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:55.081+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:55.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:54.901+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:55.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:54.901+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:55.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:54.901+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:55.896 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:55.606+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:55.896 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:55.606+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:55.896 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:55.606+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:56.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:56.084+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:56.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:56.084+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:56.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:56.084+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:56.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:55.910+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:56.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:55.910+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:56.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:55.910+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:56.903 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:56.558+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:56.903 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:56.558+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:56.903 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:56.558+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:57.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:57.038+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:57.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:57.038+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:57.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:57.038+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:57.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:56.917+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:57.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:56.917+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:57.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:56.917+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:57.896 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:57.536+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:57.896 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:57.536+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:57.896 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:57.536+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:58.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:58.030+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:58.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:58.030+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:58.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:58.030+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:58.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:57.910+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:58.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:57.910+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:58.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:57.910+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:58.859 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:58.518+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:58.859 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:58.518+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:58.859 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:58.518+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:59.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:59.020+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:59.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:59.020+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:59.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:51:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:51:59.020+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:51:59.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:58.873+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:59.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:58.873+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:59.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:58.873+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:51:59.823 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:59.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:59.823 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:59.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:51:59.823 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:51:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:51:59.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:00.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:00.048+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:00.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:00.048+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:00.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:00.048+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:00.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:59.837+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:00.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:59.837+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:00.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:51:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:51:59.837+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:00.782 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:00.544+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:00.783 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:00.544+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:00.783 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:00.544+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:01.068 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:00.796+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:01.068 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:00.796+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:01.068 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:00.796+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:01.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:01.082+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:01.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:01.082+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:01.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:01.082+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:01.742 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:01.497+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:01.743 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:01.497+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:01.743 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:01.497+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:02.001 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:01.757+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:02.002 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:01.757+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:02.002 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:01.757+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:02.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:02.057+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:02.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:02.057+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:02.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:02.057+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:02.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:02.739+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:02.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:02.739+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:02.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:02.739+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:02.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:02.518+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:02.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:02.518+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:02.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:02.518+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:03.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:03.079+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:03.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:03.079+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:03.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:03.079+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:03.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:03.564+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:03.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:03.564+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:03.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:03.564+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:03.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:03.710+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:03.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:03.710+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:03.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:03.710+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:04.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:04.109+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:04.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:04.109+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:04.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:04.109+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:04.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:04.671+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:04.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:04.671+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:04.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:04.671+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:04.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:04.579+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:04.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:04.579+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:04.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:04.579+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:05.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:05.093+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:05.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:05.093+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:05.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:05.093+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:05.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:05.541+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:05.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:05.541+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:05.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:05.541+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:05.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:05.650+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:05.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:05.650+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:05.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:05.650+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:06.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:06.072+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:06.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:06.072+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:06.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:06.072+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:06.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:06.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:06.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:06.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:06.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:06.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:06.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:06.602+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:06.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:06.602+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:06.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:06.602+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:07.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:07.118+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:07.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:07.118+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:07.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:07.118+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:07.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:07.509+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:07.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:07.509+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:07.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:07.509+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:07.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:07.576+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:07.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:07.576+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:07.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:07.576+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:08.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:08.139+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:08.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:08.139+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:08.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:08.139+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:08.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:08.540+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:08.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:08.540+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:08.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:08.540+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:08.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:08.568+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:08.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:08.568+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:08.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:08.568+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:09.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:09.095+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:09.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:09.095+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:09.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:09.095+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:09.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:09.497+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:09.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:09.497+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:09.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:09.497+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:09.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:09.585+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:09.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:09.585+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:09.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:09.585+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:10.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:10.081+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:10.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:10.081+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:10.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:10.081+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:10.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:10.484+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:10.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:10.484+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:10.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:10.484+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:10.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:10.591+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:10.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:10.591+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:10.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:10.591+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:11.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:11.042+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:11.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:11.042+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:11.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:11.042+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:11.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:11.470+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:11.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:11.470+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:11.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:11.470+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:11.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:11.618+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:11.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:11.618+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:11.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:11.618+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:12.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:12.070+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:12.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:12.070+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:12.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:12.070+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:12.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:12.460+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:12.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:12.460+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:12.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:12.460+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:12.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:12.659+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:12.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:12.659+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:12.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:12.659+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:13.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:13.033+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:13.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:13.033+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:13.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:13.033+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:13.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:13.452+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:13.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:13.452+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:13.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:13.452+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:13.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:13.637+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:13.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:13.637+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:13.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:13.637+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:14.423 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:14.047+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:14.423 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:14.047+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:14.423 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:14.047+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:14.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:14.437+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:14.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:14.437+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:14.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:14.437+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:14.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:14.649+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:14.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:14.649+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:14.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:14.649+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:15.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:15.023+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:15.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:15.023+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:15.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:15.023+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:15.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:15.450+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:15.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:15.450+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:15.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:15.450+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:15.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:15.687+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:15.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:15.687+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:15.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:15.687+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:16.400 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:15.997+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:16.400 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:15.997+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:16.401 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:15.997+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:16.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:16.415+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:16.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:16.415+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:16.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:16.415+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:17.016 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:16.732+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:17.016 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:16.732+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:17.016 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:16.732+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:17.403 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:17.030+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:17.403 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:17.030+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:17.403 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:17.030+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:17.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:17.417+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:17.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:17.417+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:17.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:17.417+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:18.113 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:17.996+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:18.113 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:17.996+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:18.113 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:17.996+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:18.113 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:17.774+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:18.113 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:17.774+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:18.113 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:17.774+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:18.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:18.433+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:18.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:18.433+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:18.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:18.433+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:19.047 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:18.968+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:19.047 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:18.968+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:19.047 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:18.968+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:19.047 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:18.752+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:19.047 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:18.752+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:19.047 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:18.752+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:19.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:19.385+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:19.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:19.385+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:19.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:19.385+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:19.986 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:19.750+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:19.986 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:19.750+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:19.986 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:19.750+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:19.986 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:19.999+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:19.986 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:19.999+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:19.986 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:19.999+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:20.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:20.429+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:20.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:20.429+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:20.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:20.429+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:21.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:21.005+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:21.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:21.005+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:21.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:21.005+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:21.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:20.788+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:21.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:20.788+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:21.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:20.788+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:21.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:21.437+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:21.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:21.437+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:21.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:21.437+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:22.001 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:21.753+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:22.001 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:21.753+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:22.001 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:21.753+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:22.396 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:22.042+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:22.397 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:22.042+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:22.397 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:22.042+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:22.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:22.411+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:22.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:22.411+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:22.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:22.411+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:22.995 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:22.745+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:22.995 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:22.745+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:22.995 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:22.745+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:23.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:23.010+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:23.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:23.010+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:23.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:23.010+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:23.734 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:23.445+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:23.734 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:23.445+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:23.735 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:23.445+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:24.015 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:23.749+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:24.015 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:23.749+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:24.016 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:23.749+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:24.422 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:24.029+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:24.422 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:24.029+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:24.422 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:24.029+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:24.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:24.465+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:24.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:24.465+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:24.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:24.465+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:25.065 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:24.777+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:25.065 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:24.777+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:25.065 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:24.777+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:25.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:25.079+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:25.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:25.079+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:25.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:25.079+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:25.780 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:25.480+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:25.780 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:25.480+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:25.780 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:25.480+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:26.071 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:25.795+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:26.072 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:25.795+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:26.072 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:25.795+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:26.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:26.086+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:26.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:26.086+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:26.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:26.086+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:26.806 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:26.484+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:26.806 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:26.484+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:26.806 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:26.484+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:27.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:27.068+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:27.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:27.068+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:27.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:27.068+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:27.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:26.820+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:27.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:26.820+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:27.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:26.820+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:27.775 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:27.454+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:27.775 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:27.454+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:27.775 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:27.454+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:28.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:28.026+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:28.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:28.026+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:28.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:28.026+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:28.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:27.789+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:28.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:27.789+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:28.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:27.789+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:28.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:28.464+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:28.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:28.464+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:28.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:28.464+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:29.003 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:28.766+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:29.004 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:28.766+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:29.004 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:28.766+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:29.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:29.018+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:29.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:29.018+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:29.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:29.018+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:29.769 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:29.473+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:29.769 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:29.473+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:29.769 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:29.473+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:30.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:30.009+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:30.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:30.009+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:30.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:30.009+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:30.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:29.784+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:30.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:29.784+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:30.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:29.784+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:30.803 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:30.490+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:30.803 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:30.490+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:30.803 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:30.490+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:31.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:31.046+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:31.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:31.046+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:31.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:31.046+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:31.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:30.817+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:31.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:30.817+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:31.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:30.817+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:31.774 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:31.451+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:31.775 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:31.451+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:31.775 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:31.451+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:32.056 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:31.788+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:32.056 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:31.788+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:32.056 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:31.788+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:32.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:32.070+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:32.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:32.070+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:32.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:32.070+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:32.753 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:32.477+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:32.753 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:32.477+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:32.753 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:32.477+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:33.060 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:32.768+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:33.061 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:32.768+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:33.061 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:32.768+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:33.420 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:33.075+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:33.420 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:33.075+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:33.420 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:33.075+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:33.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:33.434+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:33.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:33.434+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:33.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:33.434+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:34.052 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:33.742+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:34.053 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:33.742+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:34.053 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:33.742+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:34.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:34.066+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:34.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:34.066+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:34.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:34.066+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:34.765 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:34.458+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:34.765 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:34.458+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:34.766 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:34.458+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:35.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:34.779+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:35.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:34.779+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:35.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:34.779+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:35.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:35.024+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:35.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:35.024+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:35.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:35.024+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:35.764 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:35.505+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:35.764 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:35.505+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:35.764 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:35.505+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:36.013 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:35.777+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:36.013 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:35.777+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:36.014 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:35.777+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:36.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:36.027+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:36.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:36.027+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:36.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:36.027+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:36.754 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:36.469+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:36.754 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:36.469+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:36.754 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:36.469+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:37.006 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:36.768+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:37.006 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:36.768+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:37.006 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:36.768+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:37.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:37.020+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:37.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:37.020+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:37.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:37.020+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:37.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:37.513+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:37.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:37.513+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:37.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:37.513+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:37.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:37.732+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:37.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:37.732+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:37.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:37.732+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:38.314 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:38.062+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:38.315 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:38.062+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:38.315 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:38.062+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:38.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:38.524+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:38.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:38.524+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:38.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:38.524+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:39.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:39.020+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:39.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:39.020+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:39.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:39.020+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:39.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:38.775+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:39.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:38.775+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:39.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:38.775+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:39.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:39.773+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:39.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:39.773+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:39.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:39.773+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:39.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:39.535+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:39.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:39.535+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:39.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:39.535+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:40.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:40.057+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:40.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:40.057+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:40.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:40.057+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:40.763 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:40.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:40.763 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:40.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:40.763 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:40.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:41.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:41.025+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:41.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:41.025+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:41.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:41.025+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:41.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:40.777+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:41.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:40.777+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:41.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:40.777+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:41.789 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:41.468+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:41.789 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:41.468+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:41.790 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:41.468+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:42.178 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:42.015+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:42.178 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:42.015+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:42.178 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:42.015+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:42.178 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:41.803+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:42.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:41.803+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:42.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:41.803+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:42.826 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:42.504+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:42.826 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:42.504+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:42.826 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:42.504+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:43.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:43.018+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:43.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:43.018+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:43.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:43.018+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:43.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:42.840+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:43.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:42.840+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:43.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:42.840+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:43.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:43.534+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:43.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:43.534+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:43.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:43.534+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:44.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:43.887+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:44.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:43.887+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:44.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:43.887+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:44.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:43.969+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:44.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:43.969+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:44.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:43.969+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:44.908 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:44.567+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:44.908 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:44.567+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:44.908 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:44.567+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:45.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:44.922+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:45.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:44.922+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:45.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:44.922+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:45.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:44.922+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:45.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:44.922+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:45.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:44.922+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:45.861 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:45.610+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:45.861 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:45.610+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:45.861 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:45.610+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:46.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:45.875+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:46.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:45.875+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:46.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:45.875+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:46.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:45.876+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:46.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:45.876+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:46.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:45.876+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:46.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:46.644+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:46.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:46.644+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:46.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:46.644+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:46.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:46.871+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:46.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:46.871+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:46.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:46.871+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:46.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:46.845+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:46.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:46.845+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:46.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:46.845+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:47.835 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:47.667+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:47.835 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:47.667+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:47.835 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:47.667+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:48.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:47.849+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:48.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:47.849+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:48.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:47.849+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:48.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:47.859+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:48.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:47.859+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:48.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:47.859+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:48.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:48.651+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:48.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:48.651+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:48.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:48.651+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:49.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:48.879+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:49.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:48.879+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:49.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:48.879+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:49.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:48.840+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:49.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:48.840+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:49.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:48.840+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:49.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:49.653+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:49.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:49.653+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:49.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:49.653+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:49.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:49.894+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:49.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:49.894+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:49.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:49.894+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:49.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:49.883+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:49.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:49.883+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:49.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:49.883+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:50.900 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:50.630+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:50.900 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:50.630+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:50.900 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:50.630+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:50.900 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:50.876+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:50.900 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:50.876+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:50.900 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:50.876+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:51.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:50.914+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:51.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:50.914+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:51.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:50.914+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:51.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:51.604+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:51.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:51.604+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:51.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:51.604+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:51.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:51.843+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:51.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:51.843+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:51.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:51.843+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:52.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:51.956+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:52.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:51.956+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:52.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:51.956+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:52.795 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:52.634+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:52.796 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:52.634+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:52.796 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:52.634+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:53.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:52.933+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:53.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:52.933+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:53.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:52.933+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:53.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:52.809+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:53.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:52.809+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:53.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:52.809+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:53.775 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:53.670+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:53.775 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:53.670+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:53.776 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:53.670+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:54.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:53.903+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:54.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:53.903+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:54.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:53.903+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:54.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:53.790+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:54.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:53.790+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:54.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:53.790+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:55.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:54.761+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:55.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:54.761+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:55.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:54.761+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:55.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:54.712+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:55.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:54.712+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:55.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:54.712+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:55.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:54.910+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:55.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:54.910+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:55.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:54.910+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:56.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:55.800+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:56.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:55.800+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:56.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:55.800+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:56.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:55.727+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:56.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:55.727+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:56.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:55.727+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:56.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:55.874+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:56.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:55.874+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:56.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:55.874+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:57.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:56.758+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:57.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:56.758+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:57.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:56.758+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:57.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:56.762+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:57.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:56.762+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:57.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:56.762+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:57.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:56.905+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:57.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:56.905+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:57.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:56.905+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:57.776 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:57.716+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:57.777 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:57.716+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:57.777 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:57.716+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:58.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:57.855+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:58.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:57.855+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:58.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:57.855+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:58.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:57.790+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:58.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:57.790+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:58.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:57.790+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:58.785 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:58.754+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:58.785 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:58.754+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:58.785 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:58.754+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:52:59.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:58.886+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:59.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:58.886+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:59.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:58.886+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:52:59.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:58.799+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:59.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:58.799+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:52:59.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:58.799+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:00.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:59.758+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:00.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:59.758+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:00.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:52:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:52:59.758+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:00.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:59.708+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:00.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:59.708+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:00.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:52:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:52:59.708+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:00.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:59.858+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:00.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:59.858+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:00.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:52:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:52:59.858+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:00.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:00.713+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:00.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:00.713+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:00.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:00.713+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:00.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:00.668+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:00.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:00.668+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:00.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:00.668+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:00.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:00.819+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:00.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:00.819+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:00.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:00.819+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:01.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:01.664+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:01.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:01.664+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:01.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:01.664+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:01.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:01.781+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:01.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:01.781+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:01.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:01.781+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:01.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:01.751+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:01.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:01.751+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:01.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:01.751+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:02.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:02.737+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:02.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:02.737+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:02.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:02.737+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:02.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:02.770+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:02.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:02.770+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:02.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:02.770+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:02.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:02.646+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:02.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:02.646+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:02.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:02.646+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:03.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:03.641+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:03.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:03.641+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:03.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:03.641+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:03.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:03.708+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:03.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:03.708+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:03.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:03.708+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:03.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:03.756+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:03.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:03.756+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:03.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:03.757+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:04.742 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:04.747+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:04.742 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:04.747+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:04.742 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:04.747+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:04.742 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:04.621+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:04.742 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:04.621+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:04.742 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:04.621+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:05.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:04.756+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:05.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:04.756+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:05.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:04.756+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:05.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:05.778+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:05.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:05.778+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:05.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:05.778+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:05.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:05.791+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:05.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:05.791+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:05.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:05.791+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:05.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:05.578+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:05.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:05.578+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:05.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:05.578+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:06.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:06.554+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:06.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:06.554+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:06.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:06.554+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:06.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:06.748+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:06.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:06.748+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:06.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:06.748+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:06.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:06.793+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:06.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:06.793+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:06.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:06.793+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:07.781 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:53:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:53:07.528+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 10 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:53:07.781 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:07.513+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:07.781 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:07.513+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:07.782 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:07.513+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:07.782 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:07.761+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:07.782 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:07.761+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:07.782 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:07.761+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:08.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:07.795+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:08.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:07.795+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:08.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:07.795+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:08.764 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:08.512+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:08.764 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:08.512+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:08.764 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:08.512+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:09.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:08.794+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:09.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:08.794+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:09.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:08.794+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:09.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:08.778+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:09.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:08.778+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:09.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:08.778+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:09.732 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:09.491+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:09.732 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:09.491+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:09.732 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:09.491+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:10.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:09.802+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:10.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:09.802+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:10.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:09.802+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:10.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:09.747+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:10.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:09.747+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:10.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:09.747+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:10.748 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:10.449+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:10.749 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:10.449+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:10.749 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:10.449+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:11.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:10.763+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:11.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:10.763+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:11.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:10.763+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:11.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:10.780+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:11.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:10.780+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:11.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:10.780+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:11.719 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:11.441+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:11.719 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:11.441+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:11.719 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:11.441+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:12.000 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:11.758+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:12.000 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:11.758+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:12.001 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:11.759+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:12.001 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:11.733+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:12.001 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:11.733+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:12.001 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:11.733+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:12.723 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:53:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:53:12.529+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 22 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:53:12.723 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:12.692+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:12.724 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:12.692+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:12.724 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:12.692+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:12.724 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:12.486+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:12.724 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:12.486+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:12.724 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:12.486+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:13.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:12.737+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:13.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:12.737+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:13.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:12.737+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:13.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:13.515+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:13.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:13.515+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:13.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:13.515+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:13.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:13.757+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:13.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:13.757+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:13.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:13.757+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:13.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:13.691+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:13.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:13.691+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:13.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:13.691+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:14.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:14.486+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:14.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:14.486+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:14.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:14.486+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:14.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:14.718+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:14.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:14.718+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:14.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:14.718+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:14.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:14.663+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:14.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:14.663+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:14.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:14.663+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:15.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:15.489+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:15.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:15.489+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:15.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:15.489+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:15.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:15.706+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:15.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:15.706+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:15.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:15.706+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:15.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:15.633+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:15.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:15.633+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:15.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:15.633+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:16.775 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:16.663+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:16.775 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:16.663+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:16.775 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:16.663+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:16.775 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:16.641+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:16.775 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:16.641+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:16.776 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:16.641+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:16.776 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:16.517+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:16.776 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:16.517+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:16.776 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:16.517+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:17.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:53:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:53:17.530+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 39 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:53:17.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:17.497+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:17.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:17.497+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:17.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:17.497+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:17.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:17.669+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:17.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:17.669+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:17.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:17.669+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:17.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:17.663+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:17.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:17.663+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:17.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:17.663+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:18.697 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:18.654+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:18.698 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:18.654+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:18.698 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:18.654+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:18.698 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:18.453+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:18.698 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:18.453+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:18.698 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:18.453+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:19.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:18.711+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:19.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:18.711+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:19.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:18.712+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:19.709 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:19.615+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:19.710 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:19.615+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:19.710 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:19.615+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:19.710 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:19.464+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:19.710 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:19.464+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:19.710 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:19.464+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:20.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:19.723+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:20.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:19.723+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:20.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:19.723+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:20.729 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:20.614+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:20.730 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:20.614+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:20.730 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:20.614+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:20.730 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:20.479+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:20.730 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:20.479+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:20.730 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:20.479+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:21.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:20.744+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:21.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:20.744+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:21.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:20.744+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:21.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:21.602+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:21.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:21.602+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:21.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:21.602+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:21.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:21.429+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:21.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:21.429+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:21.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:21.429+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:22.001 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:21.754+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:22.001 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:21.754+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:22.001 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:21.754+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:22.641 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:53:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:53:22.530+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 66 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:53:22.642 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:22.398+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:22.642 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:22.398+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:22.642 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:22.398+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:22.642 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:22.610+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:22.642 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:22.610+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:22.643 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:22.610+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:22.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:22.732+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:22.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:22.732+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:22.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:22.732+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:23.640 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:23.358+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:23.640 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:23.358+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:23.640 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:23.358+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:23.782+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:23.782+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:23.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:23.782+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:23.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:23.655+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:23.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:23.655+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:23.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:23.655+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:24.621 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:24.339+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:24.621 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:24.339+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:24.621 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:24.339+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:24.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:24.795+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:24.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:24.795+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:24.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:24.795+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:24.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:24.634+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:24.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:24.634+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:24.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:24.634+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:25.608 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:25.313+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:25.609 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:25.313+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:25.609 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:25.313+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:25.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:25.750+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:25.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:25.750+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:25.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:25.750+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:25.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:25.622+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:25.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:25.622+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:25.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:25.622+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:26.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:26.584+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:26.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:26.584+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:26.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:26.584+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:26.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:26.348+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:26.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:26.348+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:26.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:26.348+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:27.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:26.717+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:27.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:26.717+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:27.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:26.717+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:27.642 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:53:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:53:27.530+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 96 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:53:27.642 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:27.562+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:27.642 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:27.562+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:27.642 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:27.562+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:27.643 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:27.374+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:27.643 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:27.374+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:27.643 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:27.374+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:27.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:27.733+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:27.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:27.733+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:27.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:27.733+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:28.575 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:28.326+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:28.575 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:28.326+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:28.575 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:28.326+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:28.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:28.684+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:28.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:28.684+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:28.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:28.684+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:28.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:28.589+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:28.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:28.589+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:28.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:28.589+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:29.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:29.580+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:29.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:29.580+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:29.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:29.580+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:29.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:29.353+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:29.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:29.353+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:29.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:29.353+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:30.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:29.698+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:30.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:29.698+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:30.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:29.698+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:30.583 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:30.330+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:30.583 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:30.330+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:30.583 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:30.330+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:30.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:30.688+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:30.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:30.688+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:30.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:30.688+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:30.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:30.597+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:30.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:30.597+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:30.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:30.597+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:31.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:31.551+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:31.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:31.551+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:31.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:31.551+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:31.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:31.344+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:31.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:31.344+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:31.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:31.344+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:32.000 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:31.716+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:32.000 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:31.716+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:32.001 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:31.716+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:32.641 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:53:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:53:32.531+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 122 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:53:32.641 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:32.366+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:32.642 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:32.366+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:32.642 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:32.366+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:32.642 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:32.598+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:32.642 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:32.598+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:32.642 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:32.598+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:32.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:32.729+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:32.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:32.729+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:32.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:32.729+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:33.597 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:33.342+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:33.597 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:33.342+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:33.597 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:33.342+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:33.874 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:33.611+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:33.874 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:33.611+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:33.874 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:33.611+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:33.874 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:33.747+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:33.874 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:33.747+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:33.874 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:33.747+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:34.594 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:34.345+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:34.594 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:34.345+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:34.595 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:34.345+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:34.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:34.754+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:34.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:34.754+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:34.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:34.754+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:34.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:34.608+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:34.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:34.608+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:34.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:34.608+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:35.548 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:35.346+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:35.548 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:35.346+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:35.548 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:35.346+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:35.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:35.780+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:35.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:35.780+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:35.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:35.780+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:35.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:35.562+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:35.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:35.562+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:35.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:35.562+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:36.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:36.587+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:36.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:36.587+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:36.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:36.587+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:36.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:36.391+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:36.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:36.391+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:36.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:36.391+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:37.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:36.795+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:37.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:36.795+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:37.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:36.795+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:37.645 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:53:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:53:37.532+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 151 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:53:37.645 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:37.385+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:37.645 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:37.385+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:37.646 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:37.385+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:37.646 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:37.606+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:37.646 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:37.606+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:37.646 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:37.606+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:37.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:37.781+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:37.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:37.781+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:37.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:37.781+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:38.631 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:38.346+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:38.631 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:38.346+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:38.632 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:38.346+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:38.924 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:38.760+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:38.924 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:38.760+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:38.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:38.760+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:38.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:38.645+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:38.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:38.645+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:38.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:38.645+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:39.627 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:39.383+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:39.628 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:39.383+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:39.628 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:39.383+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:39.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:39.641+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:39.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:39.641+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:39.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:39.641+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:39.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:39.744+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:39.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:39.744+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:39.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:39.744+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:40.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:40.647+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:40.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:40.647+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:40.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:40.647+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:40.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:40.431+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:40.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:40.431+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:40.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:40.431+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:41.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:40.768+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:41.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:40.768+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:41.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:40.768+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:41.764 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:41.476+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:41.764 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:41.476+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:41.764 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:41.476+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:41.764 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:41.684+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:41.764 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:41.684+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:41.764 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:41.684+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:42.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:41.778+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:42.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:41.778+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:42.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:41.778+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:42.757 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:53:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:53:42.533+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 181 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:53:42.757 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:42.731+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:42.757 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:42.731+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:42.757 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:42.731+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:42.757 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:42.512+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:42.758 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:42.512+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:42.758 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:42.512+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:43.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:42.771+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:43.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:42.771+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:43.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:42.771+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:43.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:43.508+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:43.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:43.508+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:43.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:43.508+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:43.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:43.756+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:43.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:43.756+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:43.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:43.756+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:43.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:43.693+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:43.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:43.693+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:43.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:43.693+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:44.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:44.541+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:44.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:44.541+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:44.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:44.541+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:44.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:44.666+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:44.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:44.666+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:44.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:44.666+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:45.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:44.741+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:45.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:44.741+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:45.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:44.741+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:45.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:45.533+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:45.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:45.533+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:45.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:45.533+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:45.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:45.750+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:45.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:45.750+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:45.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:45.750+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:45.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:45.686+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:45.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:45.686+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:45.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:45.686+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:46.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:46.583+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:46.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:46.583+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:46.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:46.583+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:46.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:46.769+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:46.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:46.769+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:46.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:46.769+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:46.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:46.721+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:46.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:46.721+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:46.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:46.721+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:47.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:53:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:53:47.533+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 211 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:53:47.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:47.540+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:47.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:47.540+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:47.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:47.540+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:47.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:47.685+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:47.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:47.685+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:47.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:47.685+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:47.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:47.734+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:47.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:47.735+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:47.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:47.735+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:48.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:48.520+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:48.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:48.520+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:48.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:48.520+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:48.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:48.743+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:48.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:48.743+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:48.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:48.743+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:48.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:48.657+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:48.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:48.657+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:48.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:48.657+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:49.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:49.714+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:49.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:49.714+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:49.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:49.714+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:49.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:49.530+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:49.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:49.530+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:49.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:49.530+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:49.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:49.633+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:49.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:49.633+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:49.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:49.633+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:50.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:50.742+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:50.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:50.742+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:50.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:50.742+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:50.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:50.529+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:50.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:50.529+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:50.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:50.529+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:50.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:50.631+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:50.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:50.631+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:50.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:50.631+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:51.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:51.779+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:51.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:51.779+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:51.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:51.779+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:51.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:51.554+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:51.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:51.554+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:51.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:51.554+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:51.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:51.594+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:51.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:51.594+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:51.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:51.594+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:52.809 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:53:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:53:52.534+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 237 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:53:52.809 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:52.552+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:52.809 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:52.552+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:52.810 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:52.552+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:52.810 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:52.577+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:52.810 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:52.577+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:52.810 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:52.577+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:53.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:52.823+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:53.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:52.823+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:53.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:52.823+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:53.812 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:53.558+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:53.812 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:53.558+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:53.812 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:53.558+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:53.812 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:53.592+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:53.812 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:53.592+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:53.812 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:53.592+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:54.075 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:53.826+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:54.075 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:53.826+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:54.075 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:53.826+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:54.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:54.800+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:54.966 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:54.800+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:54.966 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:54.800+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:54.967 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:54.585+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:54.967 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:54.585+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:54.967 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:54.585+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:54.967 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:54.639+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:54.967 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:54.639+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:54.967 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:54.639+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:55.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:55.772+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:55.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:55.772+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:55.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:55.772+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:55.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:55.599+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:55.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:55.599+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:55.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:55.599+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:55.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:55.596+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:55.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:55.596+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:55.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:55.596+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:56.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:56.818+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:56.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:56.818+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:56.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:56.818+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:56.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:56.610+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:56.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:56.610+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:56.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:56.610+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:56.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:56.576+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:56.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:56.577+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:56.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:56.577+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:57.846 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:53:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:53:57.535+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 271 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:53:57.846 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:57.575+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:57.846 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:57.575+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:57.846 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:57.575+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:57.846 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:57.544+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:57.847 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:57.544+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:57.847 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:57.544+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:58.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:57.860+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:58.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:57.860+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:58.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:57.860+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:58.842 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:58.618+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:58.843 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:58.618+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:58.843 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:58.618+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:58.843 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:58.509+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:58.843 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:58.509+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:58.843 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:58.509+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:59.125 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:58.857+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:59.125 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:58.857+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:59.125 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:58.857+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:53:59.847 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:59.657+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:59.848 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:59.657+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:59.848 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:53:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:53:59.657+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:53:59.848 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:59.523+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:59.848 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:59.523+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:53:59.848 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:53:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:53:59.523+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:00.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:59.861+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:00.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:59.861+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:00.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:53:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:53:59.861+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:00.895 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:00.640+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:00.895 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:00.640+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:00.895 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:00.640+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:00.895 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:00.546+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:00.896 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:00.546+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:00.896 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:00.546+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:01.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:00.909+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:01.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:00.909+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:01.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:00.909+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:01.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:01.629+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:01.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:01.629+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:01.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:01.629+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:01.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:01.577+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:01.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:01.577+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:01.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:01.577+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:02.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:01.947+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:02.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:01.947+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:02.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:01.947+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:02.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:54:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:54:02.536+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 301 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:54:02.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:02.570+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:02.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:02.570+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:02.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:02.570+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:02.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:02.640+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:02.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:02.640+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:02.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:02.640+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:03.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:02.953+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:03.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:02.953+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:03.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:02.953+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:03.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:03.674+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:03.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:03.674+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:03.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:03.674+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:03.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:03.620+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:03.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:03.620+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:03.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:03.620+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:04.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:03.980+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:04.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:03.980+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:04.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:03.980+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:04.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:04.716+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:04.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:04.716+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:04.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:04.716+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:04.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:04.614+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:04.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:04.614+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:04.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:04.614+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:05.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:04.952+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:05.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:04.952+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:05.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:04.952+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:05.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:05.651+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:05.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:05.651+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:05.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:05.651+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:05.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:05.745+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:05.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:05.745+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:05.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:05.745+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:06.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:05.952+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:06.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:05.952+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:06.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:05.952+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:06.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:06.737+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:06.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:06.737+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:06.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:06.737+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:06.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:06.670+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:06.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:06.670+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:06.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:06.670+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:07.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:06.977+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:07.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:06.977+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:07.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:06.977+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:07.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:54:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:54:07.537+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 331 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:54:07.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:07.728+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:07.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:07.728+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:07.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:07.728+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:07.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:07.640+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:07.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:07.640+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:07.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:07.640+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:08.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:07.942+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:08.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:07.942+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:08.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:07.942+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:08.917 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:08.761+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:08.917 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:08.761+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:08.917 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:08.761+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:08.917 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:08.635+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:08.917 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:08.635+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:08.917 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:08.635+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:09.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:08.931+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:09.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:08.931+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:09.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:08.931+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:09.904 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:09.807+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:09.904 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:09.807+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:09.904 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:09.807+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:09.905 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:09.590+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:09.905 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:09.590+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:09.905 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:09.590+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:10.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:09.919+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:10.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:09.919+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:10.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:09.919+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:10.898 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:10.591+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:10.898 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:10.591+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:10.898 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:10.591+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:10.898 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:10.798+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:10.898 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:10.798+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:10.898 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:10.798+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:11.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:10.912+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:11.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:10.912+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:11.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:10.912+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:11.859 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:11.597+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:11.859 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:11.597+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:11.859 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:11.597+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:11.860 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:11.759+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:11.860 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:11.759+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:11.860 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:11.759+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:12.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:11.874+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:12.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:11.874+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:12.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:11.874+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:12.846 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:54:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:54:12.537+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 362 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:54:12.846 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:12.777+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:12.846 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:12.777+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:12.846 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:12.777+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:12.847 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:12.607+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:12.847 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:12.607+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:12.847 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:12.607+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:13.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:12.860+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:13.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:12.860+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:13.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:12.860+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:13.870 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:13.735+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:13.870 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:13.735+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:13.870 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:13.735+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:13.871 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:13.642+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:13.871 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:13.642+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:13.871 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:13.642+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:14.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:13.884+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:14.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:13.884+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:14.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:13.884+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:14.841 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:14.755+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:14.841 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:14.755+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:14.842 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:14.755+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:14.842 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:14.603+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:14.842 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:14.603+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:14.842 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:14.603+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:15.142 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:14.855+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:15.142 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:14.855+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:15.142 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:14.855+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:15.839 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:15.776+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:15.839 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:15.776+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:15.839 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:15.776+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:15.839 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:15.557+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:15.840 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:15.557+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:15.840 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:15.557+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:16.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:15.853+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:16.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:15.853+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:16.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:15.853+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:16.825 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:16.767+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:16.825 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:16.767+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:16.825 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:16.767+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:16.825 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:16.540+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:16.825 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:16.540+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:16.825 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:16.540+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:17.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:16.839+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:17.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:16.839+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:17.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:16.839+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:17.858 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:54:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:54:17.537+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 395 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:54:17.858 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:17.513+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:17.858 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:17.513+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:17.858 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:17.513+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:17.858 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:17.513+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:17.858 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:17.760+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:17.859 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:17.760+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:17.859 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:17.760+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:17.859 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:17.761+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:18.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:17.871+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:18.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:17.871+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:18.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:17.871+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:18.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:17.873+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:18.832 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:18.492+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:18.832 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:18.492+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:18.832 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:18.492+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:18.832 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:18.492+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:18.832 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:18.712+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:18.832 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:18.712+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:18.832 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:18.712+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:18.833 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:18.712+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:19.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:18.846+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:19.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:18.846+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:19.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:18.846+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:19.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:18.846+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:19.662 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:19.461+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:19.662 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:19.461+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:19.662 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:19.461+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:19.662 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:19.461+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:19.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:19.676+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:19.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:19.676+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:19.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:19.676+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:19.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:19.676+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:19.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:19.873+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:19.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:19.873+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:19.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:19.873+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:19.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:19.873+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:20.860 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:20.719+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:20.860 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:20.719+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:20.860 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:20.719+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:20.860 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:20.720+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:20.860 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:20.475+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:20.860 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:20.475+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:20.861 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:20.475+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:20.861 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:20.476+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:21.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:20.873+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:21.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:20.873+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:21.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:20.873+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:21.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:20.874+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:21.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:21.471+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:21.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:21.471+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:21.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:21.471+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:21.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:21.471+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:22.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:21.767+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:22.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:21.767+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:22.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:21.767+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:22.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:21.767+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:22.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:21.910+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:22.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:21.910+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:22.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:21.910+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:22.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:21.911+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:22.524 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:22.437+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:22.525 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:22.437+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:22.525 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:22.437+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:22.525 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:22.438+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:22.894 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:54:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:54:22.538+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 421 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:54:22.894 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:22.762+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:22.894 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:22.762+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:22.894 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:22.762+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:22.894 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:22.762+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:23.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:22.908+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:23.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:22.908+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:23.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:22.908+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:23.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:22.908+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:23.727 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:23.478+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:23.727 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:23.478+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:23.728 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:23.478+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:23.728 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:23.479+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:24.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:23.741+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:24.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:23.741+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:24.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:23.741+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:24.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:23.742+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:24.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:23.943+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:24.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:23.943+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:24.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:23.943+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:24.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:23.943+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:24.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:24.451+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:24.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:24.451+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:24.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:24.451+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:24.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:24.452+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:25.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:24.700+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:25.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:24.700+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:25.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:24.700+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:25.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:24.700+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:25.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:24.922+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:25.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:24.922+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:25.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:24.922+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:25.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:24.922+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:25.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:25.408+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:25.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:25.408+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:25.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:25.408+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:25.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:25.408+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:26.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:25.731+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:26.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:25.731+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:26.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:25.731+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:26.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:25.732+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:26.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:25.941+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:26.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:25.941+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:26.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:25.941+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:26.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:25.941+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:26.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:26.421+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:26.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:26.421+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:26.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:26.421+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:26.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:26.421+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:27.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:26.733+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:27.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:26.733+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:27.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:26.733+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:27.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:26.733+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:27.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:26.939+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:27.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:26.939+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:27.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:26.939+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:27.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:26.939+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:27.760 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:54:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:54:27.539+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 450 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:54:27.760 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:27.456+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:27.760 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:27.456+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:27.760 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:27.456+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:27.761 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:27.456+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:28.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:27.774+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:28.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:27.774+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:28.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:27.774+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:28.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:27.775+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:28.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:27.940+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:28.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:27.940+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:28.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:27.940+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:28.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:27.940+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:28.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:28.437+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:28.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:28.437+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:28.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:28.437+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:28.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:28.437+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:29.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:28.741+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:29.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:28.741+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:29.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:28.741+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:29.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:28.741+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:29.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:28.898+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:29.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:28.898+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:29.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:28.898+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:29.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:28.899+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:29.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:29.402+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:29.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:29.402+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:29.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:29.402+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:29.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:29.403+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:30.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:29.768+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:30.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:29.768+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:30.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:29.768+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:30.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:29.768+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:30.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:29.904+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:30.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:29.904+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:30.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:29.904+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:30.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:29.904+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:30.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:30.425+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:30.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:30.425+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:30.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:30.425+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:30.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:30.425+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:31.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:30.911+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:31.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:30.911+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:31.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:30.911+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:31.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:30.911+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:31.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:30.741+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:31.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:30.741+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:31.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:30.741+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:31.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:30.741+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:31.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:31.379+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:31.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:31.379+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:31.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:31.379+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:31.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:31.379+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:32.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:31.915+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:32.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:31.915+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:32.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:31.915+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:32.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:31.915+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:32.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:31.780+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:32.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:31.780+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:32.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:31.780+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:32.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:31.780+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:32.650 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:54:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:54:32.540+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 477 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:54:32.650 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:32.348+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:32.650 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:32.348+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:32.650 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:32.348+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:32.650 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:32.348+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:32.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:32.802+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:32.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:32.802+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:32.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:32.802+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:32.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:32.802+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:32.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:32.898+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:32.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:32.898+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:32.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:32.898+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:32.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:32.899+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:33.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:33.367+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:33.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:33.367+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:33.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:33.367+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:33.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:33.367+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:34.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:33.776+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:34.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:33.776+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:34.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:33.776+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:34.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:33.777+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:34.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:33.892+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:34.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:33.892+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:34.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:33.892+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:34.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:33.893+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:34.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:34.415+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:34.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:34.415+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:34.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:34.415+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:34.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:34.415+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:35.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:34.935+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:35.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:34.935+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:35.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:34.935+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:35.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:34.935+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:35.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:34.825+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:35.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:34.825+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:35.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:34.825+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:35.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:34.825+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:35.843 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:35.460+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:35.903 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:35.460+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:35.903 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:35.460+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:35.903 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:35.460+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:36.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:35.857+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:36.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:35.857+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:36.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:35.857+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:36.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:35.857+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:36.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:35.907+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:36.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:35.907+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:36.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:35.907+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:36.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:35.907+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:36.835 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:36.447+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:36.836 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:36.447+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:36.836 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:36.447+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:36.836 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:36.447+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:37.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:36.849+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:37.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:36.849+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:37.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:36.849+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:37.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:36.850+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:37.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:36.916+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:37.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:36.916+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:37.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:36.916+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:37.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:36.916+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:37.652 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:54:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:54:37.540+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 506 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:54:37.652 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:37.398+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:37.652 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:37.398+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:37.652 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:37.398+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:37.652 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:37.399+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:37.918 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:37.850+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:37.918 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:37.850+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:37.918 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:37.850+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:37.918 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:37.850+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:38.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:37.931+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:38.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:37.931+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:38.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:37.931+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:38.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:37.932+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:38.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:38.439+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:38.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:38.439+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:38.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:38.439+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:38.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:38.439+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:39.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:38.863+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:39.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:38.863+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:39.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:38.863+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:39.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:38.864+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:39.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:38.914+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:39.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:38.914+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:39.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:38.914+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:39.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:38.915+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:39.821 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:39.481+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:39.821 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:39.481+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:39.821 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:39.481+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:39.821 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:39.481+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:40.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:39.835+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:40.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:39.835+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:40.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:39.835+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:40.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:39.835+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:40.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:39.892+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:40.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:39.892+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:40.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:39.892+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:40.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:39.893+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:40.855 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:40.490+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:40.855 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:40.490+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:40.855 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:40.490+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:40.855 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:40.490+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:41.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:40.869+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:41.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:40.869+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:41.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:40.869+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:41.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:40.869+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:41.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:40.882+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:41.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:40.882+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:41.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:40.882+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:41.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:40.883+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:41.845 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:41.477+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:41.845 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:41.477+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:41.845 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:41.477+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:41.845 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:41.477+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:42.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:41.859+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:42.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:41.859+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:42.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:41.859+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:42.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:41.859+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:42.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:41.864+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:42.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:41.864+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:42.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:41.864+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:42.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:41.865+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:42.819 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:54:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:54:42.541+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 533 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:54:42.820 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:42.473+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:42.820 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:42.473+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:42.820 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:42.473+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:42.820 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:42.473+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:43.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:42.834+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:43.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:42.834+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:43.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:42.834+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:43.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:42.834+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:43.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:42.883+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:43.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:42.883+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:43.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:42.883+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:43.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:42.883+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:43.806 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:43.469+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:43.807 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:43.469+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:43.807 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:43.469+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:43.807 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:43.470+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:44.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:43.820+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:44.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:43.820+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:44.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:43.820+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:44.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:43.820+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:44.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:43.895+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:44.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:43.895+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:44.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:43.895+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:44.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:43.895+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:44.834 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:44.464+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:44.834 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:44.464+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:44.834 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:44.464+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:44.834 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:44.464+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:45.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:44.848+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:45.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:44.848+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:45.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:44.848+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:45.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:44.849+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:45.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:44.847+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:45.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:44.847+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:45.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:44.847+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:45.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:44.848+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:45.805 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:45.444+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:45.806 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:45.444+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:45.806 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:45.444+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:45.806 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:45.444+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:46.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:45.881+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:46.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:45.881+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:46.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:45.881+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:46.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:45.881+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:46.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:45.819+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:46.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:45.819+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:46.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:45.819+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:46.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:45.820+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:46.786 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:46.492+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:46.786 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:46.492+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:46.786 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:46.492+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:46.787 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:46.492+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:47.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:46.800+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:47.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:46.800+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:47.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:46.800+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:47.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:46.801+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:47.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:46.923+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:47.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:46.923+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:47.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:46.923+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:47.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:46.923+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:47.801 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:54:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:54:47.542+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 562 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:54:47.801 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:47.469+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:47.801 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:47.469+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:47.801 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:47.469+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:47.801 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:47.469+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:48.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:47.814+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:48.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:47.814+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:48.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:47.814+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:48.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:47.815+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:48.177 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:47.890+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:48.177 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:47.890+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:48.177 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:47.890+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:48.178 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:47.890+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:48.778 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:48.509+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:48.778 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:48.509+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:48.778 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:48.509+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:48.778 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:48.509+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:49.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:48.932+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:49.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:48.932+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:49.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:48.932+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:49.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:48.932+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:49.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:48.791+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:49.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:48.791+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:49.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:48.791+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:49.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:48.792+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:49.766 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:49.515+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:49.766 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:49.515+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:49.766 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:49.515+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:49.766 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:49.515+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:50.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:49.917+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:50.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:49.917+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:50.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:49.917+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:50.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:49.917+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:50.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:49.781+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:50.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:49.781+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:50.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:49.781+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:50.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:49.781+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:50.810 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:50.535+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:50.810 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:50.535+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:50.810 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:50.535+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:50.810 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:50.535+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:51.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:50.824+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:51.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:50.824+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:51.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:50.824+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:51.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:50.824+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:51.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:50.938+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:51.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:50.938+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:51.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:50.938+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:51.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:50.938+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:51.767 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:51.494+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:51.767 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:51.494+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:51.767 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:51.494+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:51.767 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:51.494+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:52.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:51.781+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:52.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:51.781+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:52.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:51.781+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:52.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:51.781+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:52.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:51.912+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:52.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:51.912+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:52.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:51.912+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:52.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:51.913+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:52.785 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:54:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:54:52.543+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 594 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:54:52.785 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:52.513+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:52.786 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:52.513+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:52.786 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:52.513+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:52.786 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:52.513+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:53.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:52.799+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:53.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:52.799+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:53.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:52.799+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:53.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:52.800+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:53.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:52.866+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:53.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:52.866+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:53.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:52.866+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:53.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:52.866+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:53.804 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:53.497+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:53.804 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:53.497+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:53.805 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:53.497+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:53.805 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:53.497+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:54.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:53.819+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:54.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:53.819+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:54.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:53.819+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:54.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:53.819+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:54.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:53.834+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:54.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:53.834+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:54.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:53.834+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:54.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:53.834+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:54.786 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:54.533+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:54.787 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:54.533+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:54.787 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:54.533+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:54.787 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:54.534+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:55.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:54.800+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:55.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:54.800+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:55.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:54.800+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:55.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:54.801+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:55.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:54.805+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:55.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:54.805+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:55.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:54.805+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:55.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:54.805+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:55.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:55.572+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:55.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:55.572+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:55.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:55.572+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:55.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:55.572+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:55.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:55.782+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:55.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:55.782+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:55.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:55.782+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:55.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:55.782+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:55.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:55.819+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:55.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:55.819+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:55.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:55.819+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:55.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:55.820+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:56.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:56.560+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:56.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:56.560+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:56.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:56.560+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:56.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:56.560+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:56.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:56.790+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:56.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:56.790+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:56.929 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:56.790+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:56.930 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:56.791+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:56.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:56.789+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:56.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:56.789+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:56.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:56.789+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:56.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:56.789+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:57.810 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:54:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:54:57.543+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 632 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:54:57.810 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:57.533+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:57.810 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:57.533+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:57.810 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:57.533+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:57.810 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:57.534+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:58.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:57.841+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:58.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:57.841+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:58.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:57.841+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:58.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:57.841+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:58.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:57.824+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:58.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:57.824+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:58.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:57.824+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:58.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:57.824+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:58.832 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:58.552+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:58.832 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:58.552+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:58.832 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:58.552+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:58.832 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:58.552+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:59.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:58.845+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:59.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:58.845+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:59.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:58.845+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:54:59.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:58.846+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:54:59.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:58.868+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:59.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:58.868+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:59.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:58.868+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:54:59.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:58.868+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:54:59.823 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:59.563+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:59.823 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:59.563+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:59.823 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:59.563+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:54:59.823 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:54:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:54:59.563+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:00.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:59.840+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:00.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:59.840+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:00.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:59.840+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:00.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:54:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:54:59.841+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:00.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:59.837+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:00.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:59.837+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:00.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:59.837+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:00.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:54:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:54:59.837+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:00.868 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:00.822+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:00.868 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:00.822+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:00.868 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:00.822+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:00.868 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:00.822+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:00.869 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:00.575+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:00.869 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:00.575+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:00.869 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:00.575+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:00.869 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:00.575+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:01.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:00.883+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:01.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:00.883+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:01.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:00.883+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:01.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:00.883+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:01.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:01.844+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:01.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:01.844+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:01.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:01.844+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:01.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:01.844+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:01.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:01.783+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:01.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:01.783+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:01.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:01.783+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:01.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:01.783+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:01.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:01.610+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:01.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:01.610+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:01.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:01.610+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:01.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:01.610+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:02.798 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:55:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:55:02.544+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 659 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:55:02.798 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:02.560+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:02.798 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:02.560+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:02.798 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:02.560+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:02.798 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:02.560+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:03.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:02.893+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:03.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:02.893+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:03.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:02.893+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:03.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:02.893+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:03.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:02.812+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:03.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:02.812+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:03.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:02.812+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:03.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:02.812+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:03.842 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:03.545+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:03.843 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:03.545+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:03.843 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:03.545+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:03.843 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:03.545+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:04.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:03.870+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:04.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:03.870+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:04.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:03.870+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:04.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:03.870+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:04.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:03.857+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:04.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:03.857+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:04.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:03.857+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:04.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:03.857+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:04.778 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:04.549+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:04.779 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:04.549+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:04.779 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:04.549+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:04.779 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:04.549+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:05.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:04.833+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:05.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:04.833+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:05.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:04.833+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:05.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:04.834+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:05.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:04.865+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:05.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:04.865+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:05.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:04.865+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:05.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:04.865+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:05.829 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:05.499+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:05.829 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:05.499+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:05.829 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:05.499+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:05.829 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:05.499+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:06.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:05.843+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:06.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:05.843+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:06.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:05.843+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:06.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:05.843+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:06.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:05.880+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:06.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:05.880+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:06.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:05.880+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:06.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:05.880+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:06.841 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:06.512+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:06.841 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:06.512+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:06.841 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:06.512+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:06.841 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:06.512+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:07.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:06.855+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:07.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:06.855+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:07.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:06.855+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:07.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:06.855+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:07.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:06.884+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:07.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:06.884+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:07.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:06.884+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:07.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:06.884+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:07.854 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:55:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:55:07.545+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 688 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:55:07.854 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:07.468+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:07.854 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:07.468+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:07.854 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:07.468+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:07.854 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:07.468+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:08.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:07.868+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:08.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:07.868+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:08.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:07.868+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:08.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:07.868+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:08.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:07.877+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:08.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:07.877+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:08.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:07.877+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:08.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:07.877+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:08.849 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:08.460+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:08.849 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:08.460+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:08.849 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:08.460+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:08.849 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:08.460+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:09.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:08.863+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:09.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:08.863+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:09.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:08.863+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:09.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:08.863+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:09.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:08.888+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:09.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:08.888+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:09.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:08.888+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:09.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:08.888+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:09.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:09.431+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:09.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:09.431+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:09.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:09.431+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:09.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:09.431+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:10.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:09.844+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:10.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:09.844+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:10.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:09.844+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:10.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:09.844+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:10.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:09.923+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:10.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:09.923+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:10.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:09.923+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:10.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:09.924+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:10.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:10.396+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:10.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:10.396+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:10.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:10.396+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:10.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:10.396+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:11.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:10.849+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:11.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:10.849+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:11.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:10.849+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:11.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:10.849+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:11.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:10.966+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:11.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:10.966+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:11.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:10.966+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:11.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:10.967+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:11.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:11.419+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:11.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:11.419+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:11.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:11.419+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:11.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:11.419+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:12.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:11.872+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:12.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:11.872+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:12.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:11.872+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:12.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:11.872+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:12.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:12.006+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:12.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:12.006+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:12.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:12.006+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:12.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:12.007+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:12.659 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:55:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:55:12.545+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 715 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:55:12.659 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:12.407+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:12.659 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:12.407+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:12.659 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:12.407+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:12.659 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:12.407+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:12.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:12.837+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:12.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:12.837+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:12.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:12.837+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:12.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:12.837+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:13.404 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:13.044+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:13.404 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:13.044+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:13.405 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:13.044+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:13.405 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:13.044+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:13.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:13.418+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:13.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:13.418+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:13.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:13.418+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:13.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:13.419+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:14.067 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:13.815+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:14.067 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:13.815+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:14.068 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:13.815+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:14.068 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:13.815+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:14.365 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:14.081+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:14.366 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:14.081+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:14.366 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:14.081+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:14.366 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:14.082+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:14.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:14.379+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:14.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:14.379+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:14.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:14.379+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:14.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:14.380+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:15.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:14.856+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:15.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:14.857+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:15.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:14.857+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:15.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:14.857+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:15.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:15.054+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:15.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:15.054+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:15.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:15.054+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:15.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:15.054+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:15.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:15.398+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:15.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:15.398+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:15.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:15.398+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:15.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:15.398+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:16.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:15.842+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:16.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:15.842+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:16.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:15.842+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:16.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:15.843+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:16.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:16.024+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:16.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:16.024+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:16.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:16.024+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:16.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:16.025+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:16.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:16.438+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:16.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:16.438+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:16.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:16.438+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:16.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:16.438+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:17.046 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:16.886+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:17.046 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:16.886+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:17.046 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:16.886+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:17.046 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:16.887+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:17.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:17.060+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:17.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:17.060+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:17.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:17.060+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:17.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:17.060+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:17.849 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:55:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:55:17.546+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 744 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:55:17.849 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:17.477+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:17.849 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:17.477+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:17.849 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:17.477+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:17.849 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:17.478+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:18.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:17.862+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:18.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:17.862+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:18.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:17.862+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:18.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:17.863+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:18.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:18.102+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:18.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:18.102+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:18.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:18.102+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:18.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:18.103+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:18.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:18.431+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:18.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:18.431+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:18.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:18.431+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:18.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:18.431+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:19.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:18.878+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:19.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:18.878+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:19.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:18.878+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:19.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:18.878+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:19.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:19.103+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:19.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:19.103+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:19.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:19.103+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:19.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:19.103+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:19.839 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:19.440+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:19.839 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:19.440+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:19.839 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:19.440+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:19.839 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:19.441+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:20.093 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:19.853+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:20.094 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:19.853+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:20.094 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:19.853+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:20.094 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:19.853+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:20.385 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:20.107+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:20.386 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:20.107+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:20.386 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:20.107+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:20.386 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:20.108+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:20.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:20.399+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:20.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:20.399+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:20.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:20.399+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:20.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:20.400+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:21.093 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:20.808+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:21.093 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:20.808+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:21.093 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:20.808+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:21.093 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:20.808+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:21.364 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:21.106+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:21.364 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:21.106+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:21.364 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:21.106+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:21.364 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:21.107+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:21.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:21.378+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:21.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:21.378+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:21.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:21.378+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:21.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:21.378+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:22.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:21.841+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:22.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:21.841+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:22.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:21.841+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:22.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:21.841+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:22.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:22.087+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:22.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:22.087+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:22.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:22.087+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:22.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:22.087+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:22.533 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:22.414+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:22.533 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:22.414+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:22.533 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:22.414+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:22.533 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:22.414+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:22.853 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:55:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:55:22.547+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 771 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:55:23.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:22.867+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:23.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:22.867+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:23.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:22.867+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:23.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:22.867+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:23.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:23.102+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:23.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:23.102+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:23.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:23.102+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:23.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:23.103+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:23.804 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:23.459+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:23.804 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:23.459+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:23.804 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:23.459+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:23.804 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:23.459+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:24.105 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:23.818+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:24.105 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:23.818+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:24.105 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:23.818+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:24.105 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:23.818+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:24.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:24.119+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:24.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:24.119+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:24.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:24.119+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:24.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:24.119+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:24.786 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:24.499+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:24.786 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:24.499+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:24.786 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:24.499+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:24.786 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:24.499+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:25.077 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:24.800+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:25.077 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:24.800+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:25.077 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:24.800+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:25.077 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:24.800+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:25.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:25.091+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:25.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:25.091+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:25.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:25.091+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:25.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:25.091+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:25.828 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:25.451+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:25.828 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:25.451+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:25.828 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:25.451+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:25.828 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:25.451+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:26.103 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:25.842+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:26.104 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:25.842+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:26.104 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:25.842+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:26.104 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:25.843+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:26.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:26.118+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:26.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:26.118+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:26.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:26.118+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:26.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:26.118+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:26.870 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:26.468+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:26.870 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:26.468+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:26.870 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:26.468+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:26.870 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:26.468+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:27.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:26.884+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:27.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:26.884+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:27.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:26.884+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:27.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:26.884+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:27.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:27.123+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:27.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:27.123+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:27.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:27.123+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:27.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:27.123+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:27.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:55:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:55:27.547+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 800 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:55:27.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:27.425+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:27.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:27.425+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:27.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:27.425+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:27.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:27.425+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:28.147 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:27.876+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:28.147 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:27.876+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:28.147 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:27.876+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:28.147 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:27.876+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:28.397 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:28.160+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:28.398 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:28.160+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:28.398 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:28.160+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:28.398 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:28.161+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:28.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:28.411+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:28.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:28.411+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:28.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:28.411+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:28.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:28.412+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:29.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:28.872+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:29.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:28.872+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:29.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:28.872+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:29.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:28.872+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:29.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:29.189+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:29.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:29.189+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:29.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:29.189+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:29.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:29.189+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:29.899 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:29.442+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:29.900 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:29.442+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:29.900 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:29.442+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:29.900 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:29.443+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:30.149 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:30.163+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:30.149 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:30.163+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:30.150 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:30.163+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:30.150 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:29.913+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:30.150 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:29.913+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:30.150 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:29.913+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:30.150 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:29.913+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:30.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:30.163+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:30.908 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:30.448+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:30.908 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:30.448+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:30.908 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:30.448+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:30.908 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:30.448+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:31.163 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:30.921+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:31.164 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:30.921+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:31.164 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:30.921+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:31.164 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:30.921+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:31.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:31.178+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:31.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:31.178+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:31.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:31.178+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:31.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:31.178+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:31.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:31.403+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:31.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:31.403+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:31.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:31.403+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:31.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:31.403+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:32.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:31.938+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:32.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:31.938+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:32.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:31.938+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:32.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:31.938+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:32.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:32.153+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:32.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:32.153+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:32.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:32.153+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:32.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:32.153+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:32.913 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:55:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:55:32.548+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 827 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:55:32.913 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:32.440+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:32.913 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:32.440+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:32.913 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:32.440+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:32.913 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:32.440+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:33.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:32.927+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:33.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:32.927+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:33.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:32.927+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:33.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:32.927+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:33.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:33.163+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:33.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:33.163+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:33.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:33.163+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:33.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:33.164+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:33.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:33.461+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:33.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:33.461+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:33.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:33.461+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:33.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:33.462+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:34.399 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:34.137+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:34.399 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:34.137+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:34.399 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:34.137+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:34.399 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:34.138+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:34.400 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:33.944+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:34.400 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:33.944+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:34.400 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:33.945+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:34.400 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:33.945+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:34.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:34.413+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:34.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:34.413+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:34.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:34.413+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:34.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:34.413+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:35.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:35.102+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:35.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:35.102+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:35.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:35.102+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:35.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:35.102+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:35.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:34.904+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:35.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:34.904+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:35.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:34.904+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:35.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:34.904+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:35.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:35.400+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:35.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:35.400+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:35.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:35.400+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:35.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:35.400+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:36.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:35.906+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:36.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:35.906+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:36.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:35.906+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:36.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:35.906+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:36.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:36.130+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:36.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:36.130+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:36.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:36.130+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:36.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:36.130+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:36.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:36.429+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:36.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:36.429+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:36.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:36.429+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:36.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:36.429+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:37.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:36.893+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:37.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:36.893+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:37.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:36.893+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:37.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:36.893+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:37.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:37.106+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:37.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:37.106+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:37.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:37.106+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:37.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:37.106+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:37.886 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:55:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:55:37.549+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 856 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:55:37.887 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:37.441+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:37.887 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:37.441+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:37.887 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:37.441+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:37.887 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:37.441+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:38.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:37.900+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:38.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:37.900+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:38.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:37.900+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:38.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:37.901+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:38.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:38.069+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:38.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:38.069+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:38.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:38.069+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:38.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:38.069+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:38.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:38.449+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:38.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:38.449+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:38.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:38.449+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:38.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:38.449+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:39.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:38.906+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:39.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:38.906+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:39.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:38.906+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:39.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:38.906+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:39.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:39.044+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:39.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:39.044+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:39.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:39.044+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:39.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:39.045+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:39.900 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:39.477+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:39.900 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:39.477+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:39.900 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:39.477+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:39.900 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:39.477+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:40.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:40.053+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:40.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:40.053+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:40.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:40.053+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:40.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:40.054+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:40.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:39.914+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:40.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:39.914+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:40.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:39.914+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:40.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:39.914+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:40.863 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:40.484+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:40.863 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:40.484+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:40.863 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:40.484+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:40.863 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:40.484+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:41.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:40.877+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:41.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:40.877+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:41.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:40.877+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:41.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:40.877+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:41.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:41.013+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:41.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:41.013+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:41.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:41.013+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:41.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:41.013+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:41.903 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:41.531+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:41.903 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:41.531+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:41.903 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:41.531+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:41.903 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:41.531+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:42.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:41.917+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:42.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:41.917+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:42.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:41.917+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:42.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:41.917+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:42.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:42.043+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:42.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:42.043+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:42.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:42.043+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:42.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:42.043+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:42.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:55:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:55:42.550+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 883 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:55:42.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:42.546+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:42.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:42.546+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:42.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:42.546+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:42.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:42.546+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:43.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:42.953+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:43.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:42.953+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:43.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:42.953+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:43.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:42.953+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:43.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:43.082+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:43.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:43.082+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:43.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:43.082+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:43.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:43.082+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:43.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:43.553+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:43.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:43.553+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:43.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:43.553+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:43.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:43.553+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:44.422 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:44.107+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:44.422 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:44.107+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:44.422 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:44.107+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:44.422 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:44.107+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:44.423 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:44.001+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:44.423 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:44.001+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:44.423 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:44.001+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:44.423 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:44.001+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:44.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:44.546+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:44.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:44.546+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:44.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:44.546+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:44.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:44.546+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:45.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:45.061+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:45.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:45.061+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:45.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:45.061+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:45.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:45.062+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:45.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:44.982+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:45.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:44.982+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:45.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:44.982+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:45.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:44.982+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:45.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:45.555+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:45.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:45.555+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:45.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:45.555+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:45.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:45.555+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:46.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:46.056+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:46.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:46.056+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:46.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:46.056+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:46.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:46.056+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:46.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:45.950+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:46.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:45.950+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:46.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:45.950+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:46.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:45.950+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:46.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:46.581+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:46.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:46.581+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:46.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:46.581+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:46.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:46.581+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:47.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:47.007+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:47.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:47.007+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:47.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:47.007+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:47.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:47.007+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:47.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:46.998+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:47.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:46.998+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:47.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:46.998+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:47.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:46.998+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:47.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:55:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:55:47.551+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 912 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:55:47.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:47.587+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:47.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:47.587+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:47.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:47.587+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:47.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:47.588+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:48.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:47.988+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:48.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:47.988+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:48.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:47.988+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:48.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:47.988+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:48.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:47.964+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:48.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:47.964+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:48.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:47.964+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:48.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:47.964+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:48.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:48.576+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:48.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:48.576+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:48.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:48.576+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:48.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:48.576+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:49.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:49.024+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:49.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:49.024+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:49.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:49.024+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:49.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:49.024+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:49.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:48.976+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:49.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:48.976+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:49.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:48.976+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:49.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:48.976+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:49.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:49.561+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:49.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:49.561+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:49.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:49.561+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:49.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:49.561+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:50.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:50.049+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:50.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:50.049+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:50.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:50.049+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:50.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:50.049+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:50.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:50.008+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:50.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:50.008+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:50.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:50.008+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:50.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:50.008+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:50.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:50.594+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:50.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:50.594+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:50.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:50.594+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:50.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:50.595+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:51.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:51.031+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:51.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:51.031+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:51.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:51.031+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:51.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:51.031+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:51.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:51.002+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:51.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:51.002+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:51.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:51.002+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:51.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:51.002+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:51.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:51.632+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:51.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:51.632+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:51.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:51.632+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:51.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:51.632+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:52.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:52.073+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:52.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:52.073+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:52.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:52.073+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:52.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:52.073+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:52.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:52.051+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:52.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:52.051+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:52.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:52.051+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:52.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:52.051+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:52.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:55:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:55:52.552+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 939 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:55:52.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:52.620+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:52.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:52.620+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:52.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:52.620+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:52.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:52.620+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:53.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:53.044+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:53.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:53.044+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:53.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:53.044+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:53.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:53.044+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:53.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:53.114+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:53.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:53.114+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:53.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:53.114+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:53.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:53.114+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:53.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:53.576+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:53.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:53.576+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:53.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:53.576+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:53.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:53.576+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:54.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:54.103+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:54.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:54.103+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:54.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:54.103+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:54.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:54.103+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:54.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:54.067+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:54.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:54.067+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:54.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:54.067+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:54.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:54.067+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:54.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:54.623+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:54.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:54.623+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:54.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:54.623+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:54.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:54.623+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:55.282 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:55.042+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:55.282 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:55.042+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:55.282 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:55.042+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:55.282 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:55.042+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:55.283 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:55.062+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:55.283 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:55.062+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:55.283 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:55.062+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:55.283 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:55.062+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:55.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:55.574+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:55.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:55.574+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:55.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:55.574+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:55.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:55.574+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:56.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:56.027+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:56.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:56.027+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:56.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:56.027+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:56.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:56.027+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:56.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:56.071+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:56.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:56.071+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:56.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:56.071+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:56.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:56.071+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:56.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:56.568+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:56.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:56.568+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:56.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:56.568+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:56.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:56.568+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:57.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:57.027+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:57.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:57.027+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:57.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:57.027+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:57.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:57.027+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:57.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:57.095+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:57.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:57.095+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:57.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:57.095+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:57.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:57.095+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:57.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:55:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:55:57.552+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 968 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:55:57.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:57.567+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:57.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:57.567+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:57.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:57.567+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:57.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:57.568+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:58.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:58.007+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:58.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:58.007+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:58.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:58.007+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:58.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:58.008+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:58.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:58.075+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:58.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:58.075+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:58.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:58.076+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:58.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:58.076+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:58.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:58.519+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:58.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:58.519+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:58.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:58.519+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:58.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:58.520+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:59.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:59.057+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:59.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:59.057+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:59.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:59.057+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:55:59.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:55:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:55:59.057+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:55:59.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:59.092+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:59.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:59.092+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:59.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:59.092+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:55:59.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:55:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:55:59.092+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:55:59.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:59.561+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:59.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:59.561+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:59.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:59.561+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:55:59.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:55:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:55:59.562+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:00.332 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:00.105+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:00.332 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:00.105+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:00.333 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:00.105+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:00.333 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:00.105+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:00.333 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:00.097+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:00.333 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:00.097+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:00.333 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:00.097+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:00.333 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:00.098+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:00.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:00.587+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:00.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:00.587+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:00.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:00.587+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:00.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:00.587+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:01.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:01.119+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:01.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:01.119+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:01.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:01.119+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:01.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:01.119+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:01.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:01.106+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:01.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:01.106+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:01.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:01.106+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:01.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:01.106+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:01.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:01.602+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:01.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:01.602+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:01.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:01.602+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:01.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:01.602+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:02.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:02.083+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:02.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:02.083+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:02.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:02.083+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:02.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:02.083+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:02.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:02.089+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:02.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:02.089+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:02.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:02.089+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:02.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:02.089+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:02.926 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:56:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:56:02.553+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 995 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:56:02.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:02.576+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:02.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:02.576+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:02.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:02.576+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:02.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:02.577+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:03.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:03.079+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:03.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:03.079+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:03.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:03.079+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:03.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:03.079+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:03.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:03.117+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:03.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:03.117+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:03.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:03.117+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:03.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:03.117+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:03.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:03.553+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:03.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:03.553+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:03.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:03.553+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:03.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:03.554+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:04.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:04.055+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:04.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:04.055+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:04.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:04.055+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:04.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:04.056+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:04.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:04.137+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:04.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:04.137+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:04.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:04.137+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:04.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:04.138+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:04.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:04.577+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:04.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:04.577+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:04.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:04.577+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:04.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:04.577+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:05.383 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:05.052+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:05.383 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:05.052+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:05.383 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:05.052+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:05.383 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:05.053+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:05.384 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:05.127+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:05.384 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:05.127+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:05.384 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:05.127+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:05.384 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:05.127+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:05.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:05.549+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:05.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:05.549+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:05.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:05.549+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:05.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:05.550+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:06.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:06.087+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:06.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:06.087+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:06.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:06.087+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:06.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:06.087+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:06.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:06.158+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:06.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:06.158+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:06.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:06.158+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:06.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:06.159+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:06.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:06.538+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:06.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:06.538+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:06.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:06.538+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:06.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:06.538+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:07.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:07.043+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:07.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:07.043+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:07.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:07.043+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:07.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:07.044+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:07.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:07.171+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:07.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:07.171+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:07.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:07.171+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:07.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:07.172+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:07.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:56:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:56:07.554+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 1083 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:56:07.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:07.508+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:07.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:07.508+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:07.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:07.508+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:07.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:07.508+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:08.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:08.066+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:08.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:08.066+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:08.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:08.066+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:08.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:08.066+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:08.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:08.218+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:08.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:08.218+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:08.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:08.218+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:08.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:08.218+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:08.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:08.549+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:08.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:08.549+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:08.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:08.549+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:08.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:08.549+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:09.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:09.023+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:09.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:09.023+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:09.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:09.023+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:09.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:09.024+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:09.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:09.226+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:09.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:09.226+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:09.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:09.226+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:09.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:09.227+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:09.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:09.511+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:09.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:09.511+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:09.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:09.511+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:09.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:09.511+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:10.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:09.997+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:10.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:09.997+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:10.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:09.997+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:10.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:09.997+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:10.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:10.201+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:10.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:10.201+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:10.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:10.201+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:10.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:10.201+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:10.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:10.538+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:10.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:10.538+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:10.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:10.538+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:10.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:10.538+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:11.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:11.040+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:11.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:11.040+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:11.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:11.040+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:11.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:11.040+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:11.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:11.166+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:11.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:11.166+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:11.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:11.166+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:11.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:11.166+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:11.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:11.564+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:11.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:11.564+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:11.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:11.564+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:11.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:11.564+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:12.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:12.019+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:12.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:12.019+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:12.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:12.019+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:12.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:12.019+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:12.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:12.166+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:12.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:12.166+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:12.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:12.166+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:12.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:12.167+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:12.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:56:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:56:12.555+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 1160 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:56:12.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:12.561+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:12.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:12.561+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:12.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:12.561+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:12.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:12.561+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:13.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:12.998+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:13.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:12.998+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:13.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:12.998+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:13.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:12.998+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:13.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:13.185+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:13.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:13.185+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:13.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:13.185+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:13.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:13.186+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:13.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:13.516+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:13.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:13.516+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:13.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:13.516+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:13.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:13.517+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:14.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:13.973+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:14.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:13.973+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:14.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:13.973+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:14.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:13.974+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:14.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:14.178+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:14.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:14.178+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:14.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:14.178+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:14.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:14.178+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:14.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:14.551+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:14.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:14.551+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:14.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:14.551+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:14.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:14.551+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:15.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:14.993+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:15.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:14.993+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:15.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:14.993+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:15.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:14.993+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:15.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:15.146+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:15.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:15.146+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:15.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:15.146+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:15.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:15.146+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:15.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:15.537+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:15.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:15.537+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:15.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:15.537+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:15.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:15.537+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:16.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:15.949+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:16.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:15.949+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:16.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:15.949+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:16.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:15.949+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:16.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:16.112+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:16.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:16.112+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:16.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:16.112+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:16.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:16.112+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:16.892 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:16.555+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:16.892 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:16.555+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:16.892 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:16.555+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:16.893 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:16.556+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:17.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:16.906+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:17.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:16.906+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:17.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:16.906+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:17.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:16.907+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:17.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:17.106+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:17.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:17.106+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:17.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:17.106+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:17.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:17.106+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:17.865 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:56:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:56:17.556+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 1239 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:56:17.866 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:17.528+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:17.866 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:17.528+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:17.866 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:17.528+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:17.866 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:17.528+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:18.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T02:56:18.024+0000 7f1e03468700 0 monclient(hunting): authenticate timed out after 300 2024-04-06T02:56:18.010 INFO:teuthology.orchestra.run.smithi105.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-04-06T02:56:18.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:18.104+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:18.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:18.104+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:18.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:18.104+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:18.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:18.104+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:18.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:17.880+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:18.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:17.880+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:18.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:17.880+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:18.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:17.880+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:18.690 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-04-06T02:56:18.893 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:18.552+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:18.893 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:18.552+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:18.893 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:18.552+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:18.893 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:18.552+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:18.910 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T02:56:19.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:18.907+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:19.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:18.907+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:19.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:18.907+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:19.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:18.908+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:19.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:19.133+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:19.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:19.133+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:19.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:19.133+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:19.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:19.133+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:19.498 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:19.512+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:19.498 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:19.512+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:19.499 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:19.512+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:19.499 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:19.513+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:20.161 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:19.900+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:20.162 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:19.900+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:20.162 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:19.900+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:20.162 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:19.900+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:20.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:20.175+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:20.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:20.175+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:20.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:20.175+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:20.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:20.176+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:20.849 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:20.473+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:20.849 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:20.473+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:20.850 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:20.473+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:20.850 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:20.473+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:21.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:20.863+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:21.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:20.863+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:21.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:20.863+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:21.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:20.864+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:21.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:21.226+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:21.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:21.226+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:21.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:21.226+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:21.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:21.226+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:21.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:21.449+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:21.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:21.449+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:21.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:21.449+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:21.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:21.449+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:22.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:21.884+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:22.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:21.884+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:22.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:21.884+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:22.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:21.884+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:22.445 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:22.266+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:22.445 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:22.266+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:22.445 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:22.266+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:22.445 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:22.266+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:22.898 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:56:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:56:22.557+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 1333 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:56:22.899 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:22.469+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:22.899 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:22.469+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:22.899 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:22.469+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:22.899 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:22.469+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:23.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:22.913+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:23.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:22.913+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:23.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:22.913+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:23.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:22.913+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:23.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:23.290+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:23.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:23.290+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:23.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:23.290+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:23.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:23.290+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:23.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:23.441+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:23.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:23.441+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:23.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:23.441+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:23.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:23.441+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:24.289 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:23.953+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:24.290 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:23.953+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:24.290 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:23.953+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:24.290 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:23.953+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:24.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:24.304+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:24.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:24.304+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:24.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:24.304+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:24.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:24.304+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:24.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:24.393+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:24.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:24.393+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:24.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:24.393+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:24.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:24.393+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:25.334 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:24.960+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:25.335 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:24.960+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:25.335 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:24.960+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:25.335 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:24.961+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:25.592 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:25.348+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:25.592 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:25.348+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:25.593 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:25.348+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:25.593 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:25.349+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:25.593 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:25.350+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:25.593 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:25.350+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:25.593 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:25.350+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:25.593 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:25.351+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:26.357 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:25.964+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:26.358 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:25.964+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:26.358 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:25.964+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:26.358 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:25.964+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:26.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:26.389+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:26.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:26.389+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:26.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:26.389+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:26.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:26.389+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:26.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:26.372+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:26.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:26.372+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:26.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:26.372+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:26.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:26.372+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:27.364 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:26.959+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:27.365 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:26.959+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:27.365 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:26.959+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:27.365 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:26.959+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:27.668 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:56:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:56:27.558+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 1429 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:56:27.669 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:27.410+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:27.669 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:27.410+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:27.669 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:27.410+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:27.669 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:27.410+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:27.669 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:27.379+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:27.669 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:27.379+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:27.669 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:27.379+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:27.670 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:27.379+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:28.381 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:27.981+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:28.381 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:27.981+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:28.382 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:27.981+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:28.382 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:27.981+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:28.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:28.395+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:28.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:28.395+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:28.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:28.395+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:28.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:28.396+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:28.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:28.421+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:28.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:28.421+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:28.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:28.421+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:28.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:28.421+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:29.388 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:28.994+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:29.389 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:28.994+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:29.389 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:28.994+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:29.389 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:28.994+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:29.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:29.445+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:29.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:29.445+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:29.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:29.445+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:29.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:29.445+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:29.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:29.403+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:29.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:29.403+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:29.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:29.403+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:29.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:29.403+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:30.401 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:30.009+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:30.401 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:30.009+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:30.401 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:30.009+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:30.401 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:30.009+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:30.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:30.415+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:30.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:30.415+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:30.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:30.415+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:30.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:30.415+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:30.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:30.444+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:30.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:30.444+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:30.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:30.444+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:30.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:30.444+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:31.378 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:31.024+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:31.379 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:31.024+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:31.379 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:31.024+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:31.379 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:31.024+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:31.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:31.392+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:31.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:31.392+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:31.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:31.392+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:31.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:31.393+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:31.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:31.464+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:31.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:31.464+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:31.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:31.464+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:31.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:31.464+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:32.409 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:32.000+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:32.410 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:32.000+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:32.410 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:32.000+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:32.410 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:32.000+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:32.669 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:56:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:56:32.559+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 1523 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:56:32.669 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:32.459+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:32.670 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:32.459+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:32.670 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:32.459+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:32.670 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:32.459+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:32.670 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:32.424+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:32.670 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:32.424+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:32.670 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:32.424+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:32.670 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:32.424+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:33.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:33.039+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:33.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:33.039+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:33.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:33.039+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:33.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:33.040+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:33.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:33.468+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:33.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:33.468+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:33.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:33.468+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:33.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:33.469+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:33.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:33.468+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:33.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:33.468+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:33.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:33.468+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:33.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:33.468+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:34.405 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:34.049+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:34.406 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:34.049+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:34.406 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:34.049+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:34.406 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:34.049+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:34.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:34.419+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:34.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:34.419+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:34.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:34.419+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:34.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:34.420+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:34.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:34.460+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:34.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:34.460+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:34.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:34.460+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:34.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:34.460+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:35.385 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:35.062+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:35.385 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:35.062+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:35.385 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:35.062+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:35.386 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:35.063+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:35.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:35.468+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:35.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:35.468+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:35.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:35.468+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:35.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:35.468+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:35.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:35.400+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:35.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:35.400+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:35.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:35.400+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:35.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:35.400+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:36.420 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:36.065+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:36.420 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:36.065+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:36.420 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:36.065+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:36.420 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:36.065+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:36.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:36.434+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:36.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:36.434+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:36.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:36.434+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:36.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:36.434+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:36.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:36.465+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:36.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:36.465+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:36.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:36.465+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:36.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:36.465+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:37.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:37.080+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:37.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:37.080+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:37.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:37.080+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:37.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:37.080+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:37.451 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:37.429+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:37.451 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:37.429+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:37.451 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:37.429+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:37.451 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:37.429+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:37.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:56:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:56:37.560+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 1619 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:56:37.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:37.465+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:37.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:37.465+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:37.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:37.465+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:37.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:37.465+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:38.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:38.062+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:38.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:38.062+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:38.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:38.062+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:38.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:38.062+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:38.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:38.496+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:38.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:38.496+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:38.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:38.496+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:38.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:38.496+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:38.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:38.474+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:38.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:38.474+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:38.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:38.474+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:38.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:38.474+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:39.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:39.099+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:39.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:39.099+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:39.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:39.099+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:39.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:39.099+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:39.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:39.473+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:39.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:39.473+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:39.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:39.473+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:39.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:39.473+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:39.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:39.441+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:39.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:39.441+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:39.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:39.441+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:39.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:39.442+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:40.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:40.113+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:40.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:40.113+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:40.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:40.113+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:40.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:40.113+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:40.740 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:40.479+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:40.740 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:40.479+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:40.740 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:40.479+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:40.740 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:40.479+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:40.740 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:40.449+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:40.740 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:40.449+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:40.740 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:40.449+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:40.741 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:40.449+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:41.413 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:41.159+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:41.413 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:41.159+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:41.413 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:41.159+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:41.413 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:41.159+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:41.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:41.441+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:41.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:41.441+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:41.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:41.441+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:41.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:41.441+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:41.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:41.428+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:41.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:41.428+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:41.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:41.428+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:41.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:41.428+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:42.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:42.133+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:42.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:42.133+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:42.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:42.133+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:42.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:42.133+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:42.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:56:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:56:42.560+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 1713 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:56:42.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:42.443+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:42.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:42.444+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:42.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:42.444+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:42.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:42.444+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:42.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:42.451+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:42.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:42.451+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:42.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:42.452+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:42.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:42.452+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:43.390 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:43.147+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:43.390 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:43.147+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:43.390 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:43.147+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:43.390 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:43.147+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:43.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:43.459+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:43.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:43.459+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:43.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:43.459+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:43.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:43.459+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:43.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:43.404+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:43.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:43.404+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:43.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:43.404+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:43.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:43.404+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:44.383 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:44.098+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:44.384 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:44.098+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:44.384 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:44.098+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:44.384 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:44.098+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:44.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:44.469+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:44.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:44.469+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:44.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:44.469+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:44.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:44.469+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:44.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:44.398+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:44.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:44.398+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:44.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:44.398+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:44.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:44.398+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:45.408 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:45.063+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:45.408 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:45.063+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:45.408 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:45.063+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:45.408 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:45.063+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:45.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:45.501+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:45.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:45.501+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:45.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:45.501+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:45.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:45.501+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:45.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:45.421+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:45.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:45.421+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:45.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:45.421+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:45.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:45.421+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:46.403 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:46.084+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:46.404 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:46.084+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:46.404 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:46.084+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:46.404 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:46.084+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:46.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:46.480+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:46.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:46.480+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:46.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:46.480+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:46.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:46.480+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:46.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:46.417+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:46.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:46.417+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:46.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:46.417+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:46.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:46.417+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:47.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:47.073+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:47.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:47.073+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:47.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:47.073+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:47.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:47.073+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:47.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:56:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:56:47.561+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 1809 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:56:47.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:47.476+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:47.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:47.476+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:47.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:47.476+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:47.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:47.476+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:47.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:47.442+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:47.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:47.442+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:47.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:47.442+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:47.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:47.442+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:48.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:48.091+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:48.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:48.091+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:48.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:48.091+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:48.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:48.091+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:48.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:48.499+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:48.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:48.499+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:48.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:48.499+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:48.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:48.499+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:48.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:48.458+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:48.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:48.458+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:48.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:48.458+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:48.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:48.458+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:49.422 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:49.076+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:49.422 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:49.076+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:49.422 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:49.076+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:49.422 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:49.076+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:49.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:49.508+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:49.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:49.508+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:49.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:49.508+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:49.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:49.509+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:49.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:49.436+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:49.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:49.436+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:49.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:49.436+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:49.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:49.436+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:50.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:50.054+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:50.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:50.054+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:50.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:50.054+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:50.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:50.055+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:50.838 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:50.442+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:50.838 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:50.442+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:50.839 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:50.442+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:50.839 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:50.442+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:50.839 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:50.492+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:50.839 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:50.492+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:50.839 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:50.492+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:50.839 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:50.493+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:51.178 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:51.065+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:51.178 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:51.065+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:51.178 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:51.065+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:51.178 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:51.066+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:51.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:51.466+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:51.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:51.466+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:51.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:51.466+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:51.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:51.466+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:51.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:51.437+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:51.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:51.437+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:51.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:51.437+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:51.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:51.437+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:52.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:52.074+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:52.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:52.074+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:52.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:52.074+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:52.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:52.075+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:52.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:56:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:56:52.562+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 1903 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:56:52.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:52.506+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:52.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:52.506+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:52.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:52.506+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:52.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:52.506+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:52.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:52.441+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:52.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:52.441+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:52.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:52.441+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:52.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:52.442+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:53.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:53.101+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:53.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:53.101+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:53.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:53.101+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:53.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:53.101+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:53.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:53.486+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:53.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:53.486+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:53.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:53.486+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:53.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:53.486+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:53.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:53.475+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:53.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:53.475+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:53.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:53.475+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:53.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:53.475+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:54.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:54.067+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:54.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:54.067+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:54.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:54.067+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:54.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:54.068+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:54.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:54.452+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:54.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:54.452+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:54.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:54.452+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:54.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:54.453+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:54.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:54.524+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:54.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:54.524+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:54.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:54.524+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:54.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:54.524+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:55.270 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:55.024+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:55.270 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:55.024+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:55.270 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:55.024+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:55.270 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:55.024+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:55.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:55.502+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:55.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:55.502+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:55.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:55.502+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:55.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:55.503+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:55.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:55.493+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:55.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:55.493+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:55.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:55.493+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:55.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:55.493+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:56.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:55.985+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:56.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:55.985+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:56.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:55.985+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:56.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:55.985+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:56.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:56.538+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:56.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:56.538+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:56.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:56.538+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:56.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:56.538+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:56.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:56.501+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:56.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:56.501+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:56.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:56.501+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:56.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:56.501+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:57.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:57.015+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:57.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:57.015+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:57.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:57.015+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:57.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:57.015+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:57.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:56:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:56:57.563+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 1998 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:56:57.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:57.531+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:57.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:57.531+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:57.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:57.531+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:57.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:57.531+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:57.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:57.490+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:57.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:57.490+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:57.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:57.490+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:57.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:57.491+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:58.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:58.010+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:58.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:58.010+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:58.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:58.010+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:58.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:58.011+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:58.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:58.510+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:58.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:58.510+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:58.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:58.510+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:58.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:58.511+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:58.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:58.534+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:58.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:58.534+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:58.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:58.534+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:58.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:58.534+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:59.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:59.061+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:59.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:59.061+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:59.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:59.061+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:56:59.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:56:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:56:59.061+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:56:59.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:59.479+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:59.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:59.479+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:59.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:59.479+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:56:59.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:56:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:56:59.480+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:56:59.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:59.541+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:59.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:59.541+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:59.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:59.541+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:56:59.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:56:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:56:59.541+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:00.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:00.030+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:00.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:00.030+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:00.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:00.030+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:00.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:00.030+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:00.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:00.508+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:00.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:00.508+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:00.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:00.508+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:00.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:00.509+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:00.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:00.509+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:00.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:00.509+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:00.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:00.509+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:00.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:00.510+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:01.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:01.018+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:01.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:01.018+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:01.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:01.018+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:01.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:01.018+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:01.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:01.508+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:01.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:01.508+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:01.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:01.508+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:01.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:01.508+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:01.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:01.487+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:01.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:01.487+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:01.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:01.487+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:01.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:01.488+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:02.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:02.049+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:02.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:02.049+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:02.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:02.049+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:02.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:02.049+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:02.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:57:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:57:02.564+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 2093 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:57:02.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:02.497+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:02.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:02.497+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:02.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:02.497+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:02.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:02.497+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:02.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:02.447+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:02.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:02.447+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:02.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:02.447+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:02.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:02.447+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:03.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:03.045+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:03.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:03.046+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:03.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:03.046+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:03.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:03.046+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:03.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:03.512+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:03.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:03.512+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:03.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:03.512+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:03.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:03.513+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:03.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:03.458+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:03.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:03.458+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:03.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:03.458+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:03.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:03.458+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:04.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:04.057+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:04.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:04.057+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:04.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:04.057+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:04.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:04.057+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:04.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:04.474+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:04.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:04.474+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:04.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:04.474+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:04.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:04.475+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:04.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:04.498+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:04.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:04.498+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:04.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:04.498+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:04.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:04.499+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:05.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:05.029+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:05.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:05.029+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:05.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:05.029+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:05.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:05.030+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:05.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:05.496+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:05.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:05.496+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:05.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:05.496+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:05.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:05.497+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:05.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:05.468+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:05.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:05.468+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:05.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:05.468+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:05.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:05.469+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:06.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:06.039+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:06.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:06.039+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:06.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:06.039+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:06.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:06.039+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:06.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:06.466+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:06.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:06.466+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:06.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:06.466+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:06.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:06.466+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:06.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:06.471+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:06.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:06.471+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:06.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:06.471+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:06.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:06.472+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:07.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:07.023+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:07.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:07.024+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:07.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:07.024+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:07.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:07.024+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:07.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:57:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:57:07.565+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 2189 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:57:07.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:07.457+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:07.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:07.457+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:07.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:07.457+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:07.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:07.457+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:07.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:07.475+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:07.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:07.475+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:07.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:07.475+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:07.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:07.475+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:08.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:08.010+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:08.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:08.010+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:08.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:08.010+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:08.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:08.010+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:08.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:08.464+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:08.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:08.464+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:08.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:08.464+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:08.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:08.464+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:08.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:08.463+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:08.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:08.463+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:08.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:08.463+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:08.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:08.464+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:09.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:08.982+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:09.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:08.982+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:09.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:08.982+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:09.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:08.982+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:09.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:09.498+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:09.986 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:09.498+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:09.987 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:09.498+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:09.987 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:09.498+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:09.987 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:09.484+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:09.987 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:09.484+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:09.987 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:09.484+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:09.987 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:09.484+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:10.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:10.002+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:10.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:10.002+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:10.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:10.002+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:10.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:10.003+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:10.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:10.504+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:10.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:10.504+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:10.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:10.504+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:10.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:10.505+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:10.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:10.504+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:10.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:10.504+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:10.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:10.504+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:10.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:10.504+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:11.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:10.999+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:11.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:10.999+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:11.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:10.999+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:11.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:10.999+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:11.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:11.489+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:11.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:11.489+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:11.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:11.489+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:11.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:11.489+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:11.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:11.530+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:11.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:11.530+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:11.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:11.530+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:11.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:11.530+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:12.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:11.983+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:12.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:11.983+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:12.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:11.983+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:12.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:11.983+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:12.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:57:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:57:12.566+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 2283 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:57:12.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:12.515+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:12.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:12.515+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:12.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:12.515+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:12.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:12.515+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:12.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:12.541+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:12.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:12.541+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:12.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:12.541+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:12.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:12.542+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:13.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:12.990+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:13.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:12.990+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:13.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:12.990+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:13.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:12.991+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:13.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:13.562+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:13.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:13.562+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:13.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:13.562+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:13.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:13.563+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:13.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:13.497+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:13.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:13.497+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:13.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:13.497+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:13.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:13.497+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:14.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:13.994+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:14.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:13.994+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:14.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:13.994+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:14.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:13.994+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:14.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:14.494+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:14.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:14.494+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:14.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:14.494+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:14.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:14.494+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:14.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:14.551+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:14.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:14.551+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:14.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:14.551+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:14.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:14.551+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:15.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:14.967+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:15.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:14.967+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:15.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:14.967+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:15.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:14.967+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:15.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:15.450+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:15.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:15.450+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:15.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:15.450+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:15.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:15.451+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:15.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:15.511+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:15.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:15.511+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:15.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:15.511+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:15.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:15.511+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:16.423 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:16.015+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:16.424 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:16.015+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:16.424 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:16.015+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:16.424 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:16.015+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:16.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:16.437+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:16.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:16.437+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:16.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:16.437+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:16.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:16.438+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:16.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:16.537+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:16.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:16.537+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:16.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:16.537+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:16.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:16.537+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:17.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:16.993+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:17.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:16.993+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:17.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:16.993+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:17.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:16.993+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:17.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:57:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:57:17.567+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 2379 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:57:17.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:17.400+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:17.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:17.400+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:17.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:17.400+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:17.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:17.400+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:17.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:17.574+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:17.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:17.574+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:17.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:17.574+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:17.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:17.575+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:18.357 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:17.985+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:18.358 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:17.985+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:18.358 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:17.985+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:18.358 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:17.985+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:18.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:18.531+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:18.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:18.531+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:18.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:18.531+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:18.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:18.531+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:18.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:18.372+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:18.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:18.372+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:18.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:18.372+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:18.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:18.372+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:19.376 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:18.971+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:19.376 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:18.971+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:19.377 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:18.971+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:19.377 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:18.972+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:19.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:19.501+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:19.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:19.501+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:19.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:19.501+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:19.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:19.501+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:19.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:19.390+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:19.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:19.390+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:19.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:19.390+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:19.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:19.391+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:20.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:19.960+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:20.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:19.960+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:20.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:19.960+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:20.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:19.961+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:20.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:20.481+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:20.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:20.481+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:20.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:20.481+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:20.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:20.481+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:20.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:20.397+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:20.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:20.397+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:20.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:20.397+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:20.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:20.398+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:21.421 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:21.001+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:21.422 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:21.001+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:21.422 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:21.001+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:21.422 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:21.001+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:21.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:21.517+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:21.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:21.517+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:21.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:21.517+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:21.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:21.517+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:21.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:21.435+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:21.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:21.435+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:21.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:21.435+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:21.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:21.436+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:22.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:22.000+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:22.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:22.000+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:22.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:22.000+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:22.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:22.001+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:22.678 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:57:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:57:22.568+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 2473 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:57:22.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:22.471+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:22.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:22.471+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:22.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:22.471+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:22.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:22.471+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:22.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:22.497+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:22.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:22.497+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:22.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:22.497+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:22.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:22.498+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:23.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:22.957+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:23.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:22.957+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:23.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:22.957+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:23.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:22.957+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:23.924 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:23.493+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:23.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:23.493+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:23.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:23.493+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:23.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:23.493+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:23.458+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:23.458+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:23.458+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:23.458+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:24.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:23.939+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:24.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:23.939+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:24.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:23.939+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:24.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:23.939+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:24.898 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:24.508+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:24.898 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:24.508+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:24.898 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:24.508+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:24.898 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:24.508+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:24.899 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:24.475+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:24.899 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:24.475+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:24.899 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:24.476+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:24.899 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:24.476+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:25.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:24.912+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:25.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:24.912+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:25.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:24.912+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:25.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:24.912+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:25.860 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:25.502+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:25.860 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:25.502+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:25.860 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:25.502+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:25.860 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:25.502+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:25.861 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:25.442+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:25.861 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:25.442+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:25.861 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:25.442+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:25.861 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:25.442+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:26.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:25.875+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:26.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:25.875+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:26.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:25.875+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:26.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:25.875+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:26.907 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:26.535+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:26.907 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:26.535+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:26.907 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:26.535+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:26.907 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:26.536+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:26.908 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:26.490+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:26.908 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:26.490+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:26.908 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:26.490+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:26.908 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:26.490+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:27.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:26.922+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:27.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:26.922+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:27.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:26.922+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:27.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:26.922+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:27.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:57:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:57:27.569+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 2569 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:57:27.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:27.531+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:27.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:27.531+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:27.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:27.531+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:27.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:27.531+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:27.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:27.531+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:27.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:27.531+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:27.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:27.531+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:27.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:27.531+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:28.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:27.947+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:28.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:27.947+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:28.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:27.947+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:28.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:27.947+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:28.921 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:28.573+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:28.921 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:28.573+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:28.921 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:28.573+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:28.922 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:28.573+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:28.922 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:28.516+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:28.922 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:28.516+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:28.922 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:28.516+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:28.922 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:28.517+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:29.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:28.936+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:29.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:28.936+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:29.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:28.936+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:29.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:28.936+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:29.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:29.545+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:29.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:29.545+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:29.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:29.545+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:29.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:29.545+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:29.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:29.524+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:29.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:29.524+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:29.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:29.524+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:29.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:29.524+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:30.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:29.983+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:30.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:29.983+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:30.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:29.983+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:30.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:29.983+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:30.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:30.569+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:30.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:30.569+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:30.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:30.569+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:30.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:30.569+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:30.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:30.546+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:30.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:30.546+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:30.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:30.546+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:30.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:30.546+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:31.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:31.030+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:31.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:31.030+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:31.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:31.030+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:31.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:31.030+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:31.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:31.524+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:31.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:31.524+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:31.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:31.524+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:31.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:31.524+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:31.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:31.566+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:31.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:31.566+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:31.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:31.566+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:31.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:31.566+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:32.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:32.027+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:32.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:32.027+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:32.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:32.027+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:32.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:32.027+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:32.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:57:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:57:32.570+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 2661 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:57:32.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:32.478+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:32.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:32.478+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:32.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:32.478+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:32.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:32.478+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:32.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:32.575+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:32.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:32.575+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:32.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:32.575+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:32.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:32.578+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:33.416 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:32.985+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:33.416 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:32.985+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:33.416 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:32.985+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:33.416 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:32.985+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:33.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:33.592+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:33.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:33.592+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:33.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:33.592+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:33.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:33.592+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:33.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:33.430+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:33.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:33.430+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:33.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:33.430+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:33.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:33.430+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:34.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:33.994+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:34.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:33.994+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:34.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:33.994+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:34.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:33.995+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:34.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:34.468+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:34.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:34.468+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:34.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:34.468+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:34.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:34.468+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:34.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:34.629+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:34.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:34.629+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:34.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:34.629+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:34.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:34.629+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:35.405 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:35.041+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:35.405 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:35.041+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:35.406 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:35.041+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:35.406 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:35.041+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:35.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:35.419+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:35.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:35.419+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:35.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:35.419+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:35.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:35.419+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:35.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:35.643+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:35.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:35.643+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:35.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:35.643+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:35.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:35.643+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:36.286 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:36.008+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:36.286 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:36.008+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:36.286 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:36.008+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:36.286 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:36.008+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:36.611 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:36.426+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:36.611 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:36.426+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:36.611 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:36.426+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:36.611 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:36.426+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:36.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:36.625+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:36.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:36.625+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:36.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:36.625+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:36.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:36.625+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:37.396 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:36.977+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:37.396 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:36.977+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:37.396 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:36.977+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:37.396 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:36.977+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:37.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:57:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:57:37.571+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 2758 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:57:37.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:37.409+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:37.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:37.410+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:37.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:37.410+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:37.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:37.410+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:37.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:37.610+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:37.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:37.610+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:37.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:37.610+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:37.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:37.610+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:38.278 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:37.977+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:38.279 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:37.977+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:38.279 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:37.977+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:38.279 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:37.978+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:38.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:38.455+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:38.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:38.455+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:38.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:38.455+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:38.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:38.455+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:38.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:38.626+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:38.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:38.626+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:38.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:38.626+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:38.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:38.627+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:39.407 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:39.020+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:39.408 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:39.020+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:39.408 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:39.020+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:39.408 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:39.020+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:39.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:39.661+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:39.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:39.661+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:39.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:39.661+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:39.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:39.661+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:39.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:39.422+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:39.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:39.422+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:39.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:39.422+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:39.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:39.422+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:40.372 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:40.047+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:40.372 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:40.047+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:40.373 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:40.047+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:40.373 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:40.047+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:40.669 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:40.386+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:40.670 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:40.386+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:40.670 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:40.386+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:40.670 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:40.386+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:40.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:40.683+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:40.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:40.683+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:40.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:40.683+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:40.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:40.684+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:41.336 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:41.070+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:41.336 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:41.070+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:41.337 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:41.070+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:41.337 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:41.071+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:41.641 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:41.367+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:41.641 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:41.367+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:41.641 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:41.367+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:41.641 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:41.367+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:41.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:41.655+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:41.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:41.655+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:41.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:41.655+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:41.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:41.656+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:42.360 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:42.048+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:42.360 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:42.048+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:42.360 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:42.048+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:42.360 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:42.049+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:42.660 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:57:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:57:42.572+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 2852 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:57:42.660 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:42.374+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:42.660 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:42.374+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:42.660 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:42.374+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:42.660 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:42.374+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:42.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:42.674+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:42.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:42.674+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:42.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:42.674+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:42.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:42.674+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:43.322 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:43.011+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:43.323 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:43.011+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:43.323 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:43.011+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:43.323 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:43.011+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:43.653 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:43.337+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:43.654 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:43.337+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:43.654 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:43.337+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:43.654 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:43.337+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:43.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:43.667+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:43.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:43.667+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:43.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:43.667+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:43.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:43.668+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:44.296 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:44.032+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:44.296 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:44.032+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:44.296 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:44.032+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:44.297 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:44.032+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:44.618 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:44.310+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:44.618 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:44.310+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:44.618 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:44.310+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:44.618 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:44.310+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:44.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:44.632+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:44.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:44.632+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:44.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:44.632+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:44.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:44.632+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:45.295 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:45.038+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:45.295 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:45.038+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:45.295 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:45.038+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:45.296 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:45.038+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:45.618 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:45.309+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:45.619 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:45.309+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:45.619 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:45.309+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:45.619 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:45.309+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:45.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:45.633+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:45.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:45.633+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:45.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:45.633+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:45.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:45.633+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:46.327 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:46.049+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:46.327 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:46.049+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:46.327 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:46.049+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:46.327 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:46.050+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:46.656 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:46.341+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:46.656 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:46.341+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:46.656 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:46.341+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:46.656 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:46.341+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:46.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:46.670+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:46.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:46.670+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:46.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:46.670+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:46.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:46.670+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:47.360 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:47.014+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:47.360 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:47.014+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:47.360 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:47.014+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:47.360 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:47.015+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:47.674 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:57:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:57:47.573+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 2948 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:57:47.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:47.374+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:47.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:47.374+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:47.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:47.374+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:47.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:47.374+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:47.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:47.689+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:47.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:47.689+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:47.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:47.689+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:47.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:47.689+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:48.400 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:47.987+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:48.400 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:47.987+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:48.400 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:47.987+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:48.400 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:47.987+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:48.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:48.414+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:48.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:48.414+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:48.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:48.414+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:48.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:48.414+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:49.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:48.950+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:49.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:48.950+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:49.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:48.950+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:49.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:48.950+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:49.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:48.724+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:49.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:48.724+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:49.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:48.724+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:49.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:48.724+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:49.903 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:49.677+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:49.904 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:49.677+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:49.904 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:49.677+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:49.904 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:49.677+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:49.904 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:49.451+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:49.904 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:49.452+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:49.904 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:49.452+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:49.904 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:49.452+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:50.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:49.918+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:50.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:49.918+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:50.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:49.918+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:50.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:49.918+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:50.878 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:50.690+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:50.878 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:50.690+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:50.878 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:50.690+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:50.878 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:50.691+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:50.878 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:50.458+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:50.878 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:50.458+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:50.878 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:50.458+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:50.879 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:50.458+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:51.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:50.892+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:51.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:50.892+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:51.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:50.892+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:51.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:50.892+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:51.889 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:51.643+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:51.889 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:51.643+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:51.890 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:51.643+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:51.890 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:51.643+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:51.890 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:51.452+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:51.890 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:51.452+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:51.890 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:51.452+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:51.890 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:51.452+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:52.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:51.903+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:52.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:51.903+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:52.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:51.903+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:52.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:51.904+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:52.561 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:52.424+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:52.561 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:52.424+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:52.561 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:52.424+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:52.561 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:52.424+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:52.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:57:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:57:52.574+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 3042 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:57:52.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:52.619+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:52.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:52.619+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:52.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:52.619+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:52.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:52.619+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:53.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:52.943+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:53.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:52.943+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:53.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:52.943+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:53.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:52.944+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:53.923 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:53.651+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:53.923 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:53.651+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:53.923 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:53.651+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:53.923 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:53.651+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:53.923 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:53.466+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:53.923 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:53.466+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:53.923 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:53.466+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:53.923 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:53.467+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:54.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:53.936+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:54.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:53.936+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:54.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:53.936+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:54.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:53.937+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:54.907 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:54.478+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:54.907 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:54.478+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:54.907 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:54.478+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:54.907 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:54.478+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:54.907 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:54.615+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:54.907 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:54.615+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:54.907 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:54.615+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:54.907 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:54.615+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:55.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:54.921+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:55.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:54.921+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:55.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:54.921+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:55.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:54.921+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:55.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:55.458+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:55.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:55.458+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:55.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:55.458+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:55.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:55.458+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:55.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:55.661+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:55.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:55.661+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:55.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:55.661+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:55.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:55.661+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:56.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:55.969+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:56.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:55.969+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:56.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:55.969+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:56.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:55.970+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:56.694 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:56.457+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:56.694 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:56.457+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:56.694 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:56.457+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:56.694 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:56.457+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:56.984 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:56.708+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:56.984 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:56.708+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:56.984 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:56.708+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:56.985 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:56.708+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:57.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:56.998+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:57.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:56.998+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:57.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:56.998+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:57.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:56.999+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:57.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:57:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:57:57.576+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 3140 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:57:57.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:57.693+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:57.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:57.693+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:57.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:57.693+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:57.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:57.693+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:57.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:57.450+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:57.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:57.450+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:57.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:57.450+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:57.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:57.450+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:58.399 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:57.956+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:58.399 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:57.956+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:58.399 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:57.956+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:58.399 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:57.956+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:58.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:58.653+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:58.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:58.653+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:58.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:58.653+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:58.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:58.653+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:57:58.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:58.413+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:58.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:58.413+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:58.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:58.413+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:58.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:58.413+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:59.360 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:58.962+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:59.360 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:58.962+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:59.360 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:58.962+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:57:59.360 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:58.963+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:59.646 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:59.374+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:59.647 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:59.374+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:59.647 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:59.374+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:57:59.647 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:57:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:57:59.374+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:57:59.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:59.661+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:59.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:59.661+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:59.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:59.661+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:57:59.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:57:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:57:59.661+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:00.352 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:59.976+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:00.352 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:59.976+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:00.352 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:59.976+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:00.352 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:57:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:57:59.976+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:00.646 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:00.366+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:00.647 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:00.366+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:00.647 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:00.366+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:00.647 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:00.366+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:00.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:00.660+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:00.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:00.660+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:00.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:00.660+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:00.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:00.661+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:01.320 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:00.997+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:01.320 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:00.997+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:01.320 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:00.997+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:01.321 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:00.997+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:01.664 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:01.334+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:01.664 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:01.334+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:01.664 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:01.334+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:01.664 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:01.335+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:01.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:01.678+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:01.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:01.678+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:01.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:01.678+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:01.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:01.678+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:02.289 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:02.029+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:02.290 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:02.029+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:02.290 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:02.029+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:02.290 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:02.030+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:02.562 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:02.304+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:02.563 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:02.304+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:02.563 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:02.304+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:02.563 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:02.304+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:02.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:58:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:58:02.577+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 3235 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:58:02.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:02.702+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:02.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:02.702+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:02.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:02.702+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:02.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:02.702+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:03.315 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:02.996+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:03.316 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:02.996+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:03.316 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:02.996+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:03.316 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:02.996+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:03.665 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:03.330+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:03.665 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:03.330+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:03.665 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:03.330+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:03.666 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:03.330+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:03.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:03.679+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:03.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:03.679+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:03.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:03.679+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:03.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:03.680+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:04.275 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:03.978+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:04.275 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:03.978+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:04.275 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:03.978+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:04.275 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:03.979+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:04.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:04.290+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:04.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:04.290+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:04.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:04.290+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:04.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:04.290+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:04.951 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:04.702+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:04.951 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:04.702+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:04.951 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:04.702+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:04.951 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:04.702+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:05.313 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:04.966+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:05.313 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:04.966+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:05.313 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:04.966+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:05.313 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:04.966+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:05.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:05.327+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:05.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:05.327+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:05.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:05.327+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:05.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:05.327+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:06.118 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:05.742+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:06.118 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:05.742+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:06.118 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:05.742+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:06.118 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:05.742+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:06.118 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:05.962+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:06.119 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:05.962+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:06.119 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:05.962+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:06.119 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:05.962+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:06.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:06.332+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:06.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:06.332+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:06.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:06.332+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:06.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:06.333+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:06.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:06.713+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:06.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:06.713+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:06.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:06.713+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:06.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:06.713+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:07.339 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:06.992+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:07.340 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:06.992+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:07.340 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:06.992+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:07.340 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:06.993+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:07.670 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:58:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:58:07.578+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 3332 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:58:07.670 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:07.354+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:07.671 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:07.354+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:07.671 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:07.354+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:07.671 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:07.354+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:07.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:07.685+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:07.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:07.685+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:07.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:07.685+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:07.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:07.685+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:08.380 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:07.963+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:08.380 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:07.963+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:08.380 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:07.963+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:08.380 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:07.963+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:08.630 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:08.394+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:08.630 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:08.394+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:08.630 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:08.394+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:08.630 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:08.394+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:08.630 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:08.643+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:08.631 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:08.643+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:08.631 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:08.643+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:08.631 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:08.644+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:09.418 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:08.988+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:09.418 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:08.988+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:09.419 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:08.988+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:09.419 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:08.989+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:09.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:09.433+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:09.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:09.433+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:09.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:09.433+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:09.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:09.433+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:09.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:09.664+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:09.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:09.664+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:09.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:09.664+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:09.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:09.664+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:10.386 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:10.000+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:10.387 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:10.000+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:10.387 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:10.000+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:10.387 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:10.000+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:10.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:10.401+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:10.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:10.401+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:10.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:10.401+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:10.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:10.401+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:10.993 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:10.714+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:10.994 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:10.714+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:10.994 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:10.714+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:10.994 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:10.714+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:11.344 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:11.008+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:11.344 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:11.008+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:11.344 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:11.008+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:11.344 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:11.008+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:11.634 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:11.359+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:11.635 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:11.359+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:11.635 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:11.359+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:11.635 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:11.359+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:11.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:11.754+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:11.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:11.754+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:11.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:11.754+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:11.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:11.755+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:12.262 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:12.028+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:12.263 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:12.028+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:12.263 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:12.028+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:12.263 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:12.029+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:12.565 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:12.401+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:12.565 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:12.401+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:12.565 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:12.401+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:12.565 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:12.401+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:12.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:58:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:58:12.579+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 3424 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:58:12.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:12.778+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:12.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:12.778+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:12.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:12.778+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:12.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:12.778+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:13.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:13.064+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:13.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:13.064+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:13.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:13.064+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:13.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:13.064+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:13.728 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:13.444+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:13.728 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:13.444+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:13.729 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:13.444+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:13.729 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:13.445+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:14.056 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:13.742+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:14.056 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:13.742+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:14.056 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:13.742+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:14.056 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:13.743+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:14.409 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:14.071+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:14.409 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:14.071+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:14.409 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:14.071+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:14.409 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:14.071+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:14.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:14.423+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:14.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:14.423+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:14.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:14.423+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:14.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:14.423+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:15.074 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:14.719+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:15.074 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:14.719+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:15.074 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:14.719+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:15.074 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:14.719+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:15.334 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:15.089+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:15.334 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:15.089+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:15.334 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:15.089+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:15.334 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:15.089+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:15.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:15.429+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:15.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:15.429+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:15.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:15.429+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:15.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:15.429+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:16.034 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:15.717+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:16.034 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:15.717+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:16.034 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:15.717+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:16.035 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:15.717+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:16.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:16.049+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:16.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:16.049+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:16.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:16.049+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:16.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:16.049+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:16.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:16.445+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:16.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:16.445+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:16.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:16.445+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:16.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:16.446+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:16.685 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:16.669+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:16.685 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:16.669+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:16.685 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:16.669+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:16.685 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:16.670+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:17.399 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:17.031+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:17.400 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:17.031+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:17.400 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:17.031+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:17.400 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:17.031+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:17.660 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:58:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:58:17.580+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 3522 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:58:17.660 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:17.414+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:17.660 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:17.414+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:17.661 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:17.414+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:17.661 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:17.414+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:17.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:17.675+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:17.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:17.675+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:17.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:17.675+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:17.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:17.675+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:18.365 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:18.060+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:18.366 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:18.060+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:18.366 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:18.060+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:18.366 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:18.060+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:18.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:18.380+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:18.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:18.380+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:18.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:18.380+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:18.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:18.380+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:19.071 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:18.720+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:19.071 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:18.720+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:19.071 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:18.720+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:19.071 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:18.720+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:19.378 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:19.085+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:19.378 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:19.085+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:19.378 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:19.085+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:19.378 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:19.085+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:19.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:19.393+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:19.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:19.393+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:19.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:19.393+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:19.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:19.393+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:20.086 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:19.715+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:20.089 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:19.715+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:20.089 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:19.715+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:20.089 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:19.715+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:20.396 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:20.100+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:20.396 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:20.100+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:20.397 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:20.100+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:20.397 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:20.100+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:20.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:20.411+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:20.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:20.411+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:20.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:20.411+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:20.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:20.411+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:21.129 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:20.732+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:21.129 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:20.732+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:21.129 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:20.732+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:21.129 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:20.732+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:21.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:21.144+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:21.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:21.144+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:21.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:21.144+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:21.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:21.144+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:21.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:21.392+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:21.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:21.392+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:21.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:21.392+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:21.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:21.393+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:22.001 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:21.772+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:22.001 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:21.772+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:22.001 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:21.772+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:22.001 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:21.772+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:22.345 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:22.152+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:22.345 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:22.152+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:22.345 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:22.152+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:22.345 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:22.152+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:22.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:58:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:58:22.581+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 3615 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:58:22.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:22.359+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:22.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:22.359+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:22.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:22.359+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:22.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:22.359+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:23.094 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:22.766+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:23.094 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:22.766+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:23.094 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:22.766+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:23.094 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:22.766+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:23.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:23.338+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:23.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:23.338+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:23.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:23.338+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:23.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:23.338+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:23.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:23.109+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:23.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:23.109+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:23.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:23.109+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:23.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:23.109+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:23.782+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:23.782+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:23.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:23.782+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:23.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:23.782+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:24.372 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:24.083+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:24.372 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:24.083+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:24.372 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:24.083+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:24.372 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:24.083+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:24.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:24.387+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:24.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:24.387+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:24.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:24.387+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:24.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:24.387+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:25.023 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:24.772+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:25.023 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:24.772+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:25.024 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:24.772+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:25.024 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:24.772+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:25.348 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:25.036+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:25.348 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:25.036+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:25.348 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:25.036+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:25.348 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:25.037+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:25.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:25.362+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:25.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:25.362+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:25.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:25.362+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:25.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:25.362+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:26.048 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:25.781+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:26.049 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:25.781+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:26.049 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:25.781+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:26.049 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:25.781+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:26.382 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:26.062+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:26.382 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:26.062+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:26.382 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:26.062+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:26.382 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:26.062+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:26.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:26.396+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:26.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:26.396+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:26.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:26.396+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:26.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:26.396+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:27.075 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:26.810+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:27.075 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:26.810+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:27.075 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:26.810+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:27.075 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:26.810+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:27.374 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:27.088+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:27.375 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:27.088+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:27.375 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:27.088+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:27.375 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:27.088+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:27.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:58:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:58:27.582+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 3710 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:58:27.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:27.388+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:27.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:27.388+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:27.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:27.388+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:27.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:27.389+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:28.125 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:27.814+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:28.125 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:27.814+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:28.125 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:27.814+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:28.125 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:27.814+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:28.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:28.139+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:28.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:28.139+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:28.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:28.139+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:28.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:28.139+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:28.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:28.384+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:28.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:28.384+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:28.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:28.384+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:28.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:28.384+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:29.163 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:28.764+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:29.163 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:28.764+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:29.163 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:28.764+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:29.163 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:28.765+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:29.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:29.177+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:29.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:29.177+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:29.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:29.177+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:29.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:29.178+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:29.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:29.401+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:29.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:29.401+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:29.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:29.401+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:29.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:29.401+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:30.168 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:29.729+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:30.168 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:29.729+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:30.168 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:29.729+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:30.168 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:29.730+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:30.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:30.182+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:30.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:30.182+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:30.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:30.182+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:30.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:30.182+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:30.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:30.424+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:30.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:30.424+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:30.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:30.424+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:30.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:30.424+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:30.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:30.685+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:30.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:30.685+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:30.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:30.685+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:30.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:30.685+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:31.627 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:31.218+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:31.627 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:31.218+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:31.627 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:31.218+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:31.627 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:31.218+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:31.627 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:31.386+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:31.627 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:31.386+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:31.627 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:31.386+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:31.627 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:31.386+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:31.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:31.641+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:31.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:31.641+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:31.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:31.641+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:31.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:31.641+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:32.391 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:32.230+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:32.392 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:32.230+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:32.392 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:32.230+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:32.392 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:32.230+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:32.650 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:58:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:58:32.583+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 3802 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:58:32.650 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:32.405+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:32.651 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:32.405+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:32.651 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:32.405+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:32.651 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:32.406+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:32.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:32.664+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:32.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:32.664+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:32.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:32.664+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:32.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:32.665+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:33.428 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:33.190+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:33.428 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:33.190+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:33.428 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:33.190+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:33.429 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:33.190+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:33.766 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:33.442+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:33.766 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:33.442+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:33.766 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:33.442+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:33.766 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:33.443+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:33.766 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:33.645+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:33.767 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:33.645+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:33.767 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:33.645+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:33.767 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:33.646+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:34.634 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:34.406+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:34.634 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:34.406+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:34.634 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:34.406+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:34.634 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:34.407+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:34.634 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:34.197+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:34.634 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:34.197+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:34.634 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:34.197+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:34.635 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:34.197+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:34.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:34.647+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:34.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:34.647+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:34.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:34.647+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:34.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:34.648+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:35.602 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:35.392+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:35.602 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:35.392+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:35.602 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:35.392+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:35.602 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:35.392+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:35.603 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:35.208+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:35.603 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:35.208+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:35.603 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:35.208+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:35.603 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:35.209+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:35.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:35.616+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:35.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:35.616+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:35.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:35.616+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:35.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:35.617+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:36.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:36.390+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:36.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:36.390+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:36.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:36.390+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:36.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:36.390+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:36.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:36.164+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:36.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:36.164+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:36.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:36.164+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:36.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:36.164+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:36.883 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:36.594+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:36.883 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:36.594+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:36.883 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:36.594+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:36.883 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:36.595+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:37.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:37.128+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:37.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:37.128+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:37.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:37.128+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:37.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:37.128+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:37.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:58:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:58:37.585+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 3898 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:58:37.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:37.640+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:37.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:37.640+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:37.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:37.640+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:37.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:37.641+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:37.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:37.408+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:37.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:37.408+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:37.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:37.408+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:37.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:37.408+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:38.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:38.153+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:38.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:38.153+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:38.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:38.153+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:38.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:38.154+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:38.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:38.399+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:38.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:38.399+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:38.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:38.399+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:38.427 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:38.399+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:38.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:38.652+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:38.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:38.652+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:38.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:38.652+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:38.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:38.652+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:39.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:39.120+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:39.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:39.120+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:39.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:39.120+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:39.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:39.121+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:39.911 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:39.644+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:39.911 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:39.644+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:39.911 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:39.644+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:39.911 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:39.644+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:39.911 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:39.449+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:39.911 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:39.449+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:39.911 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:39.449+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:39.912 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:39.449+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:40.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:40.146+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:40.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:40.146+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:40.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:40.146+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:40.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:40.147+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:40.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:40.688+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:40.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:40.688+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:40.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:40.688+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:40.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:40.688+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:40.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:40.482+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:40.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:40.482+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:40.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:40.482+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:40.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:40.482+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:41.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:41.126+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:41.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:41.126+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:41.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:41.126+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:41.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:41.126+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:41.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:41.692+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:41.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:41.692+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:41.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:41.692+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:41.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:41.693+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:41.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:41.531+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:41.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:41.531+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:41.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:41.531+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:41.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:41.531+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:42.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:42.136+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:42.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:42.136+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:42.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:42.136+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:42.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:42.137+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:42.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:58:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:58:42.586+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 3991 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:58:42.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:42.678+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:42.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:42.678+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:42.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:42.678+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:42.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:42.678+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:42.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:42.535+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:42.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:42.535+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:42.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:42.535+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:42.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:42.535+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:43.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:43.130+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:43.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:43.130+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:43.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:43.130+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:43.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:43.131+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:43.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:43.693+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:43.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:43.693+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:43.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:43.693+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:43.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:43.693+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:43.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:43.576+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:43.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:43.576+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:43.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:43.576+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:43.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:43.576+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:44.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:44.124+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:44.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:44.124+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:44.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:44.124+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:44.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:44.124+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:44.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:44.656+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:44.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:44.656+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:44.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:44.656+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:44.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:44.656+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:44.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:44.569+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:44.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:44.569+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:44.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:44.569+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:44.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:44.570+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:45.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:45.130+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:45.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:45.130+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:45.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:45.130+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:45.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:45.130+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:45.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:45.618+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:45.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:45.618+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:45.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:45.618+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:45.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:45.619+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:45.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:45.548+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:45.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:45.548+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:45.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:45.548+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:45.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:45.549+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:46.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:46.162+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:46.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:46.162+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:46.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:46.162+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:46.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:46.162+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:46.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:46.631+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:46.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:46.631+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:46.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:46.631+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:46.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:46.632+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:46.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:46.545+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:46.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:46.545+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:46.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:46.545+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:46.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:46.545+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:47.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:47.119+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:47.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:47.119+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:47.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:47.119+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:47.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:47.119+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:47.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:58:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:58:47.587+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 4086 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:58:47.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:47.658+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:47.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:47.658+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:47.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:47.658+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:47.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:47.658+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:47.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:47.568+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:47.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:47.568+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:47.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:47.568+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:47.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:47.568+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:48.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:48.094+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:48.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:48.094+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:48.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:48.094+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:48.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:48.094+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:48.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:48.636+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:48.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:48.636+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:48.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:48.636+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:48.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:48.636+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:48.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:48.573+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:48.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:48.573+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:48.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:48.573+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:48.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:48.573+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:49.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:49.077+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:49.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:49.077+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:49.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:49.077+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:49.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:49.077+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:49.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:49.587+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:49.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:49.587+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:49.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:49.587+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:49.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:49.587+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:49.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:49.540+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:49.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:49.540+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:49.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:49.540+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:49.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:49.540+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:50.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:50.097+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:50.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:50.097+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:50.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:50.097+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:50.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:50.097+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:50.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:50.628+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:50.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:50.628+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:50.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:50.628+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:50.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:50.628+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:50.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:50.519+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:50.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:50.519+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:50.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:50.519+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:50.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:50.519+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:51.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:51.076+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:51.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:51.076+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:51.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:51.076+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:51.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:51.076+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:51.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:51.634+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:51.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:51.634+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:51.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:51.634+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:51.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:51.635+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:51.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:51.535+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:51.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:51.535+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:51.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:51.535+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:51.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:51.536+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:52.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:52.113+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:52.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:52.113+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:52.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:52.113+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:52.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:52.113+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:52.699 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:58:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:58:52.588+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 4182 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:58:52.699 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:52.636+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:52.699 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:52.636+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:52.699 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:52.636+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:52.699 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:52.637+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:52.700 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:52.508+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:52.700 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:52.508+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:52.700 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:52.508+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:52.700 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:52.508+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:53.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:53.130+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:53.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:53.130+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:53.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:53.130+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:53.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:53.130+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:53.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:53.613+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:53.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:53.613+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:53.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:53.613+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:53.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:53.613+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:53.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:53.553+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:53.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:53.553+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:53.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:53.553+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:53.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:53.554+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:54.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:54.146+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:54.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:54.146+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:54.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:54.146+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:54.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:54.146+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:54.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:54.628+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:54.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:54.628+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:54.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:54.628+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:54.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:54.628+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:54.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:54.541+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:54.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:54.541+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:54.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:54.541+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:54.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:54.541+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:55.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:55.110+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:55.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:55.110+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:55.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:55.110+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:55.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:55.110+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:55.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:55.629+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:55.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:55.629+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:55.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:55.629+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:55.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:55.629+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:55.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:55.532+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:55.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:55.532+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:55.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:55.532+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:55.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:55.532+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:56.427 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:56.067+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:56.427 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:56.067+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:56.427 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:56.067+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:56.427 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:56.067+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:56.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:56.661+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:56.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:56.661+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:56.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:56.661+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:56.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:56.661+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:56.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:56.570+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:56.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:56.570+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:56.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:56.570+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:56.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:56.571+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:57.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:57.079+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:57.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:57.079+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:57.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:57.079+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:57.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:57.079+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:57.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:58:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:58:57.589+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 4282 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:58:57.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:57.663+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:57.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:57.663+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:57.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:57.663+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:57.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:57.663+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:57.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:57.613+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:57.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:57.613+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:57.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:57.613+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:57.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:57.614+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:58.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:58.098+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:58.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:58.098+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:58.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:58.098+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:58.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:58.098+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:58.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:58.655+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:58.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:58.655+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:58.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:58.655+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:58.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:58.656+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:58.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:58.608+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:58.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:58.608+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:58.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:58.608+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:58.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:58.608+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:59.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:59.103+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:59.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:59.103+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:59.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:59.103+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:58:59.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:58:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:58:59.103+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:58:59.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:59.657+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:59.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:59.657+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:59.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:59.657+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:58:59.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:58:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:58:59.657+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:58:59.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:59.635+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:59.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:59.635+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:59.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:59.635+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:58:59.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:58:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:58:59.635+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:00.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:00.072+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:00.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:00.072+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:00.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:00.072+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:00.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:00.072+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:00.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:00.655+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:00.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:00.655+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:00.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:00.655+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:00.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:00.655+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:00.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:00.685+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:00.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:00.685+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:00.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:00.685+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:00.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:00.685+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:01.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:01.059+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:01.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:01.059+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:01.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:01.059+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:01.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:01.059+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:01.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:01.636+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:01.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:01.636+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:01.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:01.636+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:01.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:01.636+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:01.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:01.673+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:01.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:01.673+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:01.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:01.673+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:01.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:01.673+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:02.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:02.057+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:02.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:02.057+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:02.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:02.057+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:02.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:02.057+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:02.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:59:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:59:02.591+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 4384 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:59:02.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:02.666+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:02.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:02.666+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:02.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:02.666+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:02.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:02.667+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:02.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:02.702+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:02.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:02.702+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:02.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:02.702+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:02.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:02.702+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:03.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:03.041+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:03.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:03.041+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:03.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:03.041+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:03.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:03.042+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:03.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:03.703+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:03.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:03.703+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:03.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:03.703+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:03.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:03.704+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:03.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:03.695+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:03.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:03.695+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:03.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:03.695+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:03.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:03.695+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:04.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:04.027+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:04.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:04.027+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:04.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:04.027+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:04.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:04.027+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:04.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:04.666+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:04.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:04.666+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:04.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:04.666+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:04.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:04.666+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:04.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:04.728+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:04.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:04.728+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:04.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:04.728+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:04.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:04.728+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:05.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:04.990+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:05.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:04.990+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:05.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:04.990+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:05.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:04.990+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:05.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:05.682+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:05.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:05.682+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:05.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:05.682+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:05.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:05.682+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:05.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:05.778+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:05.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:05.778+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:05.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:05.778+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:05.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:05.778+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:06.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:05.979+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:06.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:05.979+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:06.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:05.979+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:06.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:05.979+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:06.957 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:06.815+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:06.957 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:06.815+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:06.957 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:06.815+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:06.957 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:06.816+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:06.957 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:06.696+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:06.957 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:06.696+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:06.958 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:06.696+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:06.958 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:06.696+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:07.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:06.972+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:07.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:06.972+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:07.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:06.972+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:07.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:06.972+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:07.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:59:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:59:07.592+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 4484 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:59:07.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:07.714+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:07.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:07.714+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:07.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:07.714+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:07.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:07.714+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:07.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:07.838+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:07.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:07.838+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:07.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:07.838+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:07.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:07.838+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:08.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:07.944+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:08.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:07.944+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:08.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:07.944+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:08.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:07.945+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:09.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:08.908+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:09.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:08.908+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:09.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:08.908+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:09.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:08.908+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:09.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:08.700+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:09.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:08.700+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:09.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:08.700+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:09.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:08.700+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:09.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:08.797+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:09.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:08.797+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:09.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:08.797+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:09.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:08.797+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:09.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:09.900+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:09.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:09.900+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:09.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:09.900+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:09.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:09.900+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:09.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:09.665+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:09.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:09.665+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:09.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:09.665+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:09.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:09.665+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:09.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:09.825+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:09.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:09.825+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:09.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:09.825+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:09.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:09.826+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:10.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:10.872+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:10.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:10.872+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:10.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:10.872+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:10.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:10.872+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:10.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:10.660+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:10.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:10.660+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:10.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:10.660+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:10.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:10.660+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:10.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:10.869+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:10.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:10.869+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:10.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:10.869+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:10.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:10.869+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:11.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:11.858+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:11.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:11.858+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:11.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:11.858+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:11.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:11.859+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:11.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:11.668+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:11.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:11.668+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:11.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:11.668+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:11.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:11.668+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:11.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:11.864+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:11.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:11.864+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:11.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:11.864+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:11.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:11.864+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:12.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:59:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:59:12.594+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 4577 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:59:12.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:12.651+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:12.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:12.651+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:12.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:12.651+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:12.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:12.651+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:12.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:12.836+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:12.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:12.837+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:12.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:12.837+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:12.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:12.837+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:12.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:12.836+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:12.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:12.836+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:12.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:12.836+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:12.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:12.836+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:13.789 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:13.681+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:13.789 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:13.681+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:13.789 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:13.681+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:13.789 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:13.681+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:14.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:13.804+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:14.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:13.804+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:14.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:13.804+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:14.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:13.804+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:14.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:13.830+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:14.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:13.830+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:14.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:13.830+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:14.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:13.830+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:14.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:14.673+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:14.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:14.673+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:14.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:14.673+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:14.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:14.673+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:14.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:14.797+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:14.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:14.797+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:14.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:14.797+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:14.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:14.797+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:14.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:14.778+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:14.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:14.778+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:14.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:14.778+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:14.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:14.778+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:15.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:15.729+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:15.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:15.729+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:15.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:15.729+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:15.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:15.729+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:15.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:15.635+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:15.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:15.635+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:15.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:15.635+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:15.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:15.635+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:15.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:15.810+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:15.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:15.810+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:15.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:15.810+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:15.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:15.810+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:16.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:16.702+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:16.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:16.702+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:16.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:16.702+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:16.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:16.702+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:16.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:16.629+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:16.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:16.629+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:16.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:16.629+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:16.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:16.629+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:16.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:16.786+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:16.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:16.786+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:16.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:16.786+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:16.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:16.786+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:17.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:59:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:59:17.595+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 4686 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:59:17.980 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:17.766+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:17.980 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:17.766+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:17.980 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:17.766+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:17.980 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:17.767+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:17.981 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:17.658+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:17.981 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:17.658+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:17.981 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:17.658+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:17.981 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:17.659+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:17.981 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:17.655+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:17.981 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:17.655+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:17.981 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:17.655+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:17.981 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:17.656+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:18.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:18.626+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:18.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:18.626+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:18.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:18.626+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:18.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:18.627+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:18.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:18.786+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:18.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:18.786+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:18.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:18.786+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:18.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:18.787+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:18.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:18.656+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:18.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:18.656+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:18.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:18.656+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:18.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:18.657+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:19.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:19.627+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:19.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:19.627+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:19.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:19.627+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:19.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:19.627+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:19.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:19.744+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:19.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:19.744+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:19.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:19.744+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:19.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:19.744+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:19.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:19.661+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:19.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:19.661+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:19.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:19.661+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:19.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:19.662+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:20.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:20.746+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:20.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:20.746+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:20.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:20.746+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:20.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:20.746+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:20.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:20.629+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:20.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:20.629+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:20.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:20.629+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:20.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:20.630+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:20.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:20.707+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:20.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:20.707+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:20.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:20.707+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:20.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:20.707+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:21.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:21.723+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:21.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:21.723+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:21.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:21.723+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:21.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:21.723+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:21.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:21.731+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:21.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:21.732+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:21.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:21.732+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:21.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:21.732+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:21.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:21.625+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:21.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:21.625+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:21.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:21.625+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:21.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:21.626+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:22.593 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:59:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:59:22.596+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 4780 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:59:22.918 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:22.730+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:22.919 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:22.730+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:22.919 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:22.730+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:22.919 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:22.730+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:22.919 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:22.612+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:22.919 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:22.612+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:22.919 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:22.612+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:22.919 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:22.613+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:22.919 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:22.757+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:22.920 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:22.757+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:22.920 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:22.757+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:22.920 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:22.757+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:23.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:23.692+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:23.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:23.692+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:23.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:23.692+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:23.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:23.692+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:23.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:23.593+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:23.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:23.593+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:23.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:23.593+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:23.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:23.593+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:23.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:23.782+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:23.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:23.782+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:23.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:23.782+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:23.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:23.782+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:24.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:24.717+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:24.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:24.717+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:24.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:24.717+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:24.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:24.717+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:24.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:24.615+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:24.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:24.615+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:24.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:24.615+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:24.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:24.615+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:24.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:24.760+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:24.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:24.760+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:24.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:24.760+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:24.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:24.761+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:25.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:25.707+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:25.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:25.707+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:25.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:25.707+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:25.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:25.707+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:25.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:25.665+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:25.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:25.665+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:25.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:25.665+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:25.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:25.666+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:25.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:25.758+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:25.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:25.758+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:25.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:25.758+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:25.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:25.758+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:27.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:26.699+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:27.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:26.699+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:27.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:26.699+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:27.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:26.699+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:27.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:26.710+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:27.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:26.710+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:27.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:26.710+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:27.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:26.711+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:27.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:26.715+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:27.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:26.715+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:27.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:26.715+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:27.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:26.716+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:27.653 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:59:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:59:27.597+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 4876 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:59:27.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:27.667+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:27.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:27.667+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:27.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:27.667+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:27.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:27.668+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:27.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:27.677+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:27.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:27.677+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:27.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:27.677+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:27.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:27.678+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:27.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:27.667+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:27.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:27.667+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:27.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:27.667+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:27.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:27.668+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:28.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:28.714+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:28.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:28.714+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:28.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:28.714+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:28.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:28.714+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:28.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:28.686+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:28.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:28.686+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:28.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:28.686+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:28.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:28.686+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:28.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:28.631+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:28.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:28.631+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:28.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:28.631+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:28.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:28.632+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:29.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:29.665+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:29.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:29.665+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:29.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:29.665+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:29.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:29.666+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:29.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:29.725+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:29.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:29.725+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:29.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:29.725+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:29.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:29.726+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:29.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:29.589+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:29.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:29.589+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:29.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:29.589+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:29.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:29.590+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:30.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:30.685+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:30.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:30.685+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:30.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:30.685+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:30.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:30.685+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:30.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:30.659+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:30.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:30.659+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:30.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:30.659+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:30.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:30.659+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:30.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:30.577+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:30.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:30.577+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:30.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:30.577+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:30.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:30.578+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:31.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:31.535+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:31.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:31.535+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:31.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:31.535+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:31.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:31.535+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:31.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:31.637+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:31.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:31.637+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:31.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:31.637+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:31.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:31.637+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:31.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:31.662+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:31.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:31.662+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:31.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:31.662+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:31.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:31.663+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:32.707 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:59:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:59:32.598+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 4987 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:59:32.708 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:32.636+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:32.708 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:32.636+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:32.708 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:32.636+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:32.708 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:32.636+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:32.709 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:32.683+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:32.709 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:32.683+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:32.709 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:32.683+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:32.709 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:32.683+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:32.709 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:32.572+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:32.709 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:32.572+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:32.709 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:32.572+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:32.709 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:32.573+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:33.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:33.685+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:33.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:33.686+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:33.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:33.686+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:33.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:33.686+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:33.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:33.705+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:33.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:33.705+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:33.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:33.705+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:33.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:33.705+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:33.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:33.570+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:33.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:33.570+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:33.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:33.570+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:33.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:33.571+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:34.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:34.687+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:34.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:34.687+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:34.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:34.687+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:34.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:34.687+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:34.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:34.726+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:34.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:34.726+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:34.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:34.726+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:34.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:34.727+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:34.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:34.522+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:34.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:34.522+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:34.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:34.522+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:34.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:34.522+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:35.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:35.676+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:35.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:35.676+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:35.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:35.676+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:35.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:35.676+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:35.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:35.712+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:35.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:35.712+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:35.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:35.712+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:35.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:35.712+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:35.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:35.517+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:35.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:35.517+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:35.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:35.517+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:35.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:35.517+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:36.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:36.713+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:36.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:36.713+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:36.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:36.713+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:36.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:36.714+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:36.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:36.655+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:36.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:36.655+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:36.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:36.655+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:36.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:36.655+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:36.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:36.475+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:36.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:36.475+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:36.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:36.475+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:36.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:36.476+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:37.710 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:59:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:59:37.599+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 5106 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:59:37.710 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:37.681+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:37.710 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:37.681+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:37.710 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:37.681+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:37.710 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:37.681+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:37.711 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:37.456+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:37.711 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:37.456+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:37.711 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:37.456+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:37.711 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:37.457+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:38.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:37.762+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:38.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:37.762+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:38.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:37.762+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:38.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:37.762+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:38.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:38.701+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:38.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:38.701+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:38.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:38.701+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:38.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:38.701+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:38.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:38.504+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:38.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:38.504+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:38.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:38.504+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:38.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:38.505+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:38.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:38.713+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:38.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:38.713+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:38.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:38.713+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:38.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:38.713+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:39.718 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:39.686+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:39.718 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:39.686+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:39.718 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:39.686+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:39.718 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:39.686+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:39.718 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:39.462+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:39.719 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:39.462+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:39.719 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:39.462+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:39.719 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:39.462+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:40.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:39.732+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:40.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:39.732+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:40.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:39.732+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:40.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:39.732+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:40.729 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:40.681+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:40.730 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:40.681+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:40.730 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:40.681+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:40.730 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:40.681+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:40.730 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:40.481+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:40.730 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:40.481+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:40.730 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:40.481+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:40.730 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:40.481+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:41.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:40.743+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:41.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:40.743+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:41.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:40.743+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:41.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:40.744+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:41.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:41.451+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:41.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:41.451+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:41.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:41.451+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:41.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:41.451+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:42.000 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:41.727+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:42.000 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:41.727+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:42.000 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:41.727+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:42.000 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:41.727+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:42.000 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:41.731+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:42.001 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:41.731+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:42.001 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:41.731+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:42.001 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:41.731+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:42.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:59:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:59:42.601+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 5200 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:59:42.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:42.438+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:42.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:42.438+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:42.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:42.438+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:42.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:42.438+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:43.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:42.702+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:43.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:42.702+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:43.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:42.702+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:43.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:42.702+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:43.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:42.762+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:43.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:42.762+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:43.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:42.762+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:43.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:42.762+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:43.748 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:43.659+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:43.748 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:43.659+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:43.748 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:43.659+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:43.749 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:43.659+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:43.749 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:43.488+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:43.749 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:43.488+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:43.749 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:43.488+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:43.749 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:43.488+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:44.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:43.762+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:44.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:43.762+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:44.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:43.762+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:44.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:43.763+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:44.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:44.621+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:44.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:44.621+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:44.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:44.621+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:44.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:44.621+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:44.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:44.452+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:44.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:44.452+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:44.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:44.452+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:44.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:44.452+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:45.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:44.765+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:45.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:44.765+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:45.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:44.765+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:45.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:44.765+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:45.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:45.631+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:45.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:45.631+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:45.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:45.631+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:45.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:45.632+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:45.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:45.403+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:45.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:45.403+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:45.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:45.403+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:45.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:45.403+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:46.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:45.717+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:46.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:45.717+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:46.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:45.717+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:46.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:45.717+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:46.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:46.639+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:46.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:46.639+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:46.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:46.639+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:46.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:46.639+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:46.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:46.414+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:46.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:46.414+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:46.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:46.414+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:46.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:46.414+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:47.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:46.722+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:47.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:46.722+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:47.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:46.722+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:47.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:46.723+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:47.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:59:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:59:47.602+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 5296 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:59:47.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:47.406+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:47.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:47.406+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:47.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:47.406+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:47.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:47.407+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:47.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:47.602+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:47.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:47.602+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:47.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:47.602+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:47.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:47.603+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:48.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:47.710+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:48.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:47.710+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:48.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:47.710+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:48.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:47.710+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:48.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:48.601+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:48.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:48.601+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:48.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:48.601+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:48.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:48.601+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:48.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:48.693+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:48.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:48.693+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:48.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:48.693+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:48.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:48.693+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:48.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:48.448+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:48.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:48.448+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:48.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:48.448+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:48.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:48.448+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:49.654 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:49.414+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:49.654 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:49.414+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:49.654 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:49.414+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:49.654 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:49.414+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:49.654 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:49.577+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:49.655 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:49.577+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:49.655 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:49.577+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:49.655 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:49.577+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:49.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:49.668+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:49.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:49.668+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:49.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:49.668+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:49.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:49.668+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:50.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:50.601+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:50.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:50.601+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:50.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:50.601+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:50.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:50.601+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:50.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:50.675+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:50.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:50.675+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:50.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:50.675+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:50.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:50.675+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:50.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:50.461+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:50.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:50.461+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:50.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:50.461+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:50.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:50.461+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:51.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:51.559+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:51.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:51.559+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:51.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:51.559+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:51.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:51.559+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:51.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:51.664+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:51.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:51.664+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:51.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:51.664+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:51.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:51.664+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:51.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:51.456+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:51.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:51.456+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:51.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:51.456+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:51.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:51.456+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:52.716 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:59:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:59:52.603+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 5389 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:59:52.717 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:52.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:52.717 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:52.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:52.717 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:52.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:52.717 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:52.568+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:52.717 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:52.652+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:52.717 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:52.652+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:52.717 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:52.652+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:52.717 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:52.652+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:52.718 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:52.506+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:52.718 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:52.506+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:52.718 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:52.506+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:52.718 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:52.506+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:53.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:53.598+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:53.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:53.598+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:53.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:53.598+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:53.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:53.598+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:53.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:53.639+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:53.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:53.639+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:53.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:53.639+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:53.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:53.639+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:53.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:53.527+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:53.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:53.527+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:53.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:53.527+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:53.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:53.527+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:54.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:54.575+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:54.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:54.575+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:54.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:54.575+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:54.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:54.575+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:54.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:54.670+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:54.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:54.670+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:54.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:54.670+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:54.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:54.670+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:54.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:54.514+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:54.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:54.514+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:54.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:54.514+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:54.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:54.514+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:55.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:55.614+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:55.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:55.614+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:55.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:55.614+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:55.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:55.614+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:55.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:55.667+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:55.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:55.667+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:55.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:55.667+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:55.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:55.667+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:55.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:55.563+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:55.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:55.563+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:55.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:55.563+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:55.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:55.563+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:56.894 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:56.585+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:56.894 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:56.585+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:56.894 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:56.585+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:56.894 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:56.585+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:56.894 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:56.626+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:56.894 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:56.626+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:56.895 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:56.626+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:56.895 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:56.626+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:56.895 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:56.526+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:56.895 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:56.526+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:56.895 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:56.526+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:56.895 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:56.526+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:57.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 02:59:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T02:59:57.605+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 5486 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T02:59:57.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:57.521+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:57.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:57.521+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:57.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:57.521+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:57.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:57.521+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:57.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:57.607+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:57.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:57.607+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:57.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:57.607+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:57.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:57.607+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:57.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:57.555+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:57.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:57.555+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:57.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:57.555+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:57.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:57.555+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:58.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:58.531+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:58.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:58.531+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:58.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:58.531+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:58.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:58.531+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:58.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:58.655+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:58.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:58.655+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:58.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:58.655+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:58.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:58.655+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:58.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:58.552+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:58.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:58.552+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:58.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:58.552+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:58.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:58.552+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:59.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:59.581+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:59.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:59.581+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:59.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:59.581+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T02:59:59.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 02:59:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T02:59:59.581+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T02:59:59.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:59.660+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:59.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:59.660+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:59.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:59.660+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T02:59:59.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 02:59:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T02:59:59.660+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T02:59:59.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:59.505+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:59.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:59.505+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:59.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:59.505+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T02:59:59.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 02:59:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T02:59:59.505+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:00.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:00.650+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:00.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:00.650+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:00.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:00.650+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:00.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:00.650+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:00.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:00.511+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:00.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:00.511+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:00.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:00.511+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:00.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:00.511+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:00.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:00.628+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:00.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:00.628+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:00.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:00.628+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:00.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:00.628+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:01.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:01.628+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:01.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:01.628+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:01.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:01.628+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:01.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:01.628+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:01.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:01.619+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:01.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:01.619+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:01.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:01.619+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:01.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:01.619+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:01.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:01.499+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:01.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:01.499+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:01.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:01.499+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:01.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:01.500+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:02.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:00:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:00:02.605+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 5580 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:00:02.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:02.600+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:02.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:02.600+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:02.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:02.600+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:02.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:02.600+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:02.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:02.510+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:02.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:02.510+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:02.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:02.510+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:02.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:02.510+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:02.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:02.676+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:02.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:02.676+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:02.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:02.676+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:02.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:02.676+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:03.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:03.635+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:03.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:03.635+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:03.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:03.635+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:03.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:03.635+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:03.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:03.592+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:03.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:03.592+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:03.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:03.592+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:03.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:03.592+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:03.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:03.476+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:03.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:03.476+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:03.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:03.476+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:03.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:03.476+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:04.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:04.601+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:04.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:04.601+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:04.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:04.601+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:04.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:04.601+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:04.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:04.613+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:04.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:04.613+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:04.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:04.613+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:04.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:04.613+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:04.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:04.507+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:04.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:04.507+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:04.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:04.507+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:04.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:04.507+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:05.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:05.640+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:05.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:05.640+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:05.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:05.640+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:05.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:05.640+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:05.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:05.627+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:05.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:05.627+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:05.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:05.627+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:05.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:05.627+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:05.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:05.504+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:05.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:05.505+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:05.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:05.505+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:05.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:05.505+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:06.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:06.536+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:06.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:06.536+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:06.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:06.536+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:06.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:06.536+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:06.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:06.677+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:06.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:06.677+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:06.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:06.677+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:06.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:06.678+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:06.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:06.609+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:06.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:06.609+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:06.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:06.609+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:06.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:06.609+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:07.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:00:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:00:07.607+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 5675 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:00:07.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:07.570+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:07.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:07.570+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:07.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:07.570+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:07.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:07.570+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:07.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:07.587+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:07.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:07.587+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:07.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:07.587+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:07.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:07.588+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:07.928 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:07.682+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:07.928 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:07.682+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:07.928 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:07.682+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:07.928 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:07.682+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:08.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:08.676+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:08.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:08.676+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:08.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:08.676+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:08.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:08.677+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:08.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:08.610+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:08.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:08.610+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:08.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:08.610+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:08.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:08.611+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:08.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:08.589+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:08.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:08.589+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:08.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:08.589+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:08.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:08.589+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:09.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:09.675+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:09.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:09.675+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:09.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:09.675+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:09.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:09.676+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:09.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:09.603+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:09.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:09.603+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:09.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:09.603+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:09.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:09.603+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:09.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:09.540+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:09.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:09.540+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:09.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:09.540+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:09.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:09.540+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:10.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:10.668+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:10.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:10.668+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:10.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:10.668+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:10.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:10.669+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:10.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:10.612+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:10.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:10.612+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:10.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:10.612+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:10.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:10.612+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:10.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:10.556+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:10.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:10.556+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:10.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:10.556+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:10.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:10.557+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:11.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:11.660+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:11.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:11.660+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:11.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:11.660+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:11.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:11.660+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:11.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:11.629+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:11.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:11.629+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:11.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:11.629+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:11.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:11.629+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:11.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:11.535+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:11.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:11.535+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:11.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:11.535+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:11.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:11.536+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:12.834 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:00:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:00:12.609+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 5768 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:00:12.834 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:12.564+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:12.834 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:12.564+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:12.834 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:12.564+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:12.835 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:12.565+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:12.835 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:12.704+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:12.835 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:12.704+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:12.835 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:12.704+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:12.835 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:12.704+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:12.835 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:12.610+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:12.835 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:12.610+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:12.836 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:12.610+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:12.836 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:12.610+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:13.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:13.665+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:13.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:13.665+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:13.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:13.665+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:13.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:13.665+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:13.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:13.624+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:13.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:13.624+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:13.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:13.624+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:13.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:13.625+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:13.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:13.576+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:13.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:13.576+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:13.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:13.576+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:13.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:13.577+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:14.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:14.601+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:14.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:14.601+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:14.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:14.601+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:14.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:14.601+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:14.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:14.530+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:14.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:14.530+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:14.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:14.530+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:14.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:14.530+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:14.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:14.655+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:14.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:14.655+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:14.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:14.655+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:14.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:14.656+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:15.599 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:15.570+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:15.599 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:15.570+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:15.599 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:15.570+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:15.599 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:15.571+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:15.599 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:15.487+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:15.599 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:15.487+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:15.600 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:15.487+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:15.600 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:15.487+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:15.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:15.613+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:15.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:15.613+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:15.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:15.613+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:15.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:15.613+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:16.875 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:16.620+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:16.875 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:16.620+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:16.875 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:16.620+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:16.875 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:16.620+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:16.875 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:16.589+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:16.875 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:16.589+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:16.876 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:16.589+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:16.876 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:16.589+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:16.876 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:16.529+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:16.876 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:16.529+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:16.876 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:16.529+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:16.876 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:16.529+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:17.806 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:00:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:00:17.610+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 5864 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:00:17.806 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:17.600+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:17.806 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:17.600+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:17.807 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:17.600+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:17.807 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:17.600+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:17.807 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:17.633+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:17.807 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:17.633+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:17.807 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:17.633+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:17.807 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:17.633+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:17.808 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:17.510+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:17.808 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:17.510+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:17.808 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:17.510+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:17.808 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:17.510+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:18.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:18.614+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:18.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:18.614+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:18.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:18.614+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:18.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:18.615+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:18.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:18.654+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:18.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:18.654+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:18.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:18.654+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:18.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:18.654+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:18.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:18.550+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:18.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:18.550+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:18.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:18.550+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:18.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:18.551+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:19.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:19.610+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:19.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:19.610+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:19.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:19.610+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:19.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:19.610+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:19.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:19.617+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:19.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:19.617+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:19.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:19.617+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:19.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:19.618+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:19.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:19.507+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:19.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:19.507+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:19.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:19.507+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:19.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:19.507+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:20.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:20.655+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:20.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:20.655+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:20.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:20.655+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:20.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:20.656+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:20.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:20.587+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:20.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:20.587+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:20.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:20.587+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:20.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:20.588+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:20.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:20.534+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:20.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:20.534+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:20.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:20.534+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:20.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:20.534+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:21.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:21.687+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:21.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:21.687+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:21.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:21.687+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:21.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:21.687+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:21.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:21.546+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:21.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:21.546+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:21.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:21.546+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:21.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:21.546+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:21.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:21.518+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:21.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:21.518+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:21.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:21.518+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:21.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:21.518+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:22.696 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:00:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:00:22.611+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 5959 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:00:22.697 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:22.504+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:22.697 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:22.504+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:22.697 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:22.504+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:22.697 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:22.505+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:22.697 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:22.539+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:22.697 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:22.539+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:22.697 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:22.539+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:22.697 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:22.540+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:23.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:22.710+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:23.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:22.710+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:23.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:22.710+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:23.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:22.711+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:23.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:23.676+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:23.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:23.676+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:23.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:23.676+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:23.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:23.676+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:23.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:23.476+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:23.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:23.476+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:23.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:23.476+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:23.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:23.477+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:23.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:23.503+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:23.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:23.503+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:23.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:23.503+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:23.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:23.503+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:24.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:24.671+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:24.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:24.671+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:24.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:24.671+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:24.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:24.672+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:24.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:24.446+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:24.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:24.446+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:24.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:24.446+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:24.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:24.447+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:24.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:24.504+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:24.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:24.504+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:24.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:24.505+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:24.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:24.505+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:25.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:25.632+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:25.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:25.632+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:25.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:25.632+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:25.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:25.633+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:25.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:25.433+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:25.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:25.433+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:25.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:25.433+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:25.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:25.433+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:25.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:25.505+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:25.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:25.505+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:25.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:25.505+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:25.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:25.505+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:26.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:26.635+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:26.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:26.635+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:26.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:26.635+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:26.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:26.635+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:26.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:26.425+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:26.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:26.425+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:26.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:26.425+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:26.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:26.425+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:26.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:26.520+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:26.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:26.520+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:26.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:26.520+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:26.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:26.520+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:27.599 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:27.472+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:27.599 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:27.472+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:27.599 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:27.472+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:27.599 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:27.472+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:27.599 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:27.488+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:27.599 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:27.488+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:27.599 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:27.488+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:27.600 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:27.488+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:27.874 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:00:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:00:27.612+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 6054 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:00:27.874 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:27.616+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:27.874 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:27.616+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:27.874 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:27.616+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:27.874 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:27.616+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:28.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:28.633+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:28.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:28.633+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:28.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:28.633+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:28.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:28.633+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:28.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:28.483+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:28.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:28.483+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:28.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:28.483+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:28.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:28.483+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:28.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:28.481+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:28.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:28.481+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:28.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:28.481+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:28.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:28.481+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:29.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:29.619+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:29.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:29.619+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:29.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:29.619+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:29.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:29.619+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:29.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:29.442+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:29.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:29.442+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:29.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:29.442+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:29.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:29.442+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:29.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:29.514+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:29.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:29.514+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:29.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:29.514+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:29.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:29.514+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:30.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:30.623+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:30.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:30.623+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:30.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:30.623+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:30.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:30.623+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:30.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:30.501+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:30.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:30.501+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:30.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:30.501+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:30.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:30.501+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:30.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:30.437+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:30.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:30.437+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:30.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:30.437+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:30.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:30.438+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:31.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:31.669+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:31.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:31.669+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:31.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:31.669+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:31.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:31.670+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:31.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:31.535+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:31.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:31.535+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:31.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:31.535+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:31.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:31.535+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:31.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:31.450+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:31.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:31.450+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:31.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:31.450+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:31.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:31.451+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:32.726 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:00:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:00:32.614+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 6148 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:00:32.727 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:32.479+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:32.727 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:32.479+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:32.727 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:32.479+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:32.727 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:32.480+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:32.727 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:32.636+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:32.727 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:32.636+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:32.728 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:32.636+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:32.728 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:32.637+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:32.728 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:32.527+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:32.728 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:32.527+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:32.728 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:32.527+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:32.728 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:32.527+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:33.766 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:33.519+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:33.766 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:33.519+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:33.766 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:33.519+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:33.766 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:33.519+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:33.767 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:33.607+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:33.767 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:33.607+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:33.767 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:33.607+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:33.767 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:33.608+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:33.767 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:33.484+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:33.767 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:33.484+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:33.767 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:33.484+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:33.767 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:33.484+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:34.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:34.564+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:34.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:34.564+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:34.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:34.564+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:34.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:34.564+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:34.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:34.645+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:34.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:34.645+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:34.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:34.645+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:34.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:34.645+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:34.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:34.528+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:34.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:34.528+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:34.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:34.528+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:34.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:34.528+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:35.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:35.559+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:35.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:35.559+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:35.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:35.559+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:35.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:35.559+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:35.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:35.615+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:35.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:35.615+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:35.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:35.615+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:35.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:35.616+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:35.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:35.536+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:35.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:35.536+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:35.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:35.536+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:35.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:35.537+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:36.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:36.601+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:36.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:36.601+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:36.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:36.601+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:36.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:36.602+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:36.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:36.601+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:36.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:36.601+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:36.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:36.601+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:36.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:36.602+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:36.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:36.551+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:36.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:36.551+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:36.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:36.551+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:36.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:36.552+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:37.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:00:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:00:37.615+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 6244 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:00:37.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:37.566+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:37.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:37.566+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:37.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:37.566+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:37.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:37.566+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:37.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:37.565+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:37.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:37.565+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:37.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:37.565+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:37.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:37.565+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:37.929 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:37.640+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:37.929 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:37.640+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:37.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:37.640+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:37.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:37.640+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:38.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:38.552+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:38.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:38.552+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:38.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:38.552+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:38.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:38.553+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:38.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:38.664+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:38.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:38.664+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:38.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:38.664+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:38.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:38.664+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:38.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:38.559+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:38.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:38.559+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:38.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:38.559+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:38.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:38.559+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:39.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:39.535+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:39.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:39.535+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:39.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:39.535+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:39.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:39.535+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:39.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:39.580+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:39.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:39.580+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:39.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:39.580+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:39.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:39.581+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:39.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:39.643+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:39.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:39.643+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:39.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:39.643+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:39.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:39.643+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:40.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:40.521+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:40.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:40.521+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:40.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:40.521+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:40.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:40.521+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:40.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:40.669+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:40.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:40.669+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:40.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:40.669+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:40.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:40.669+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:40.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:40.617+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:40.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:40.617+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:40.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:40.617+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:40.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:40.617+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:41.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:41.532+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:41.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:41.532+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:41.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:41.532+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:41.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:41.532+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:41.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:41.588+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:41.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:41.588+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:41.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:41.588+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:41.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:41.588+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:41.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:41.635+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:41.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:41.635+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:41.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:41.635+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:41.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:41.635+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:42.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:00:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:00:42.617+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 6339 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:00:42.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:42.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:42.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:42.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:42.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:42.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:42.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:42.501+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:42.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:42.640+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:42.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:42.640+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:42.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:42.640+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:42.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:42.641+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:42.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:42.601+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:42.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:42.601+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:42.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:42.601+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:42.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:42.601+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:43.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:43.513+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:43.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:43.513+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:43.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:43.513+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:43.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:43.513+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:43.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:43.670+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:43.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:43.670+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:43.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:43.670+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:43.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:43.671+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:43.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:43.582+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:43.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:43.582+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:43.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:43.582+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:43.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:43.582+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:44.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:44.476+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:44.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:44.476+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:44.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:44.476+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:44.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:44.476+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:44.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:44.547+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:44.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:44.547+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:44.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:44.547+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:44.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:44.548+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:44.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:44.632+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:44.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:44.632+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:44.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:44.632+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:44.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:44.633+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:45.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:45.461+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:45.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:45.461+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:45.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:45.461+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:45.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:45.461+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:45.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:45.584+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:45.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:45.584+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:45.928 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:45.584+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:45.928 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:45.584+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:45.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:45.659+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:45.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:45.659+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:45.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:45.659+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:45.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:45.659+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:46.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:46.470+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:46.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:46.470+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:46.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:46.470+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:46.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:46.470+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:46.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:46.658+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:46.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:46.658+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:46.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:46.658+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:46.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:46.658+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:46.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:46.535+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:46.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:46.535+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:46.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:46.535+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:46.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:46.535+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:47.731 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:00:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:00:47.618+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 6434 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:00:47.732 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:47.479+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:47.732 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:47.479+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:47.732 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:47.479+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:47.732 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:47.479+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:47.732 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:47.515+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:47.732 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:47.515+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:47.733 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:47.515+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:47.733 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:47.515+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:47.733 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:47.684+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:47.733 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:47.684+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:47.733 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:47.684+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:47.733 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:47.685+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:48.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:48.526+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:48.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:48.526+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:48.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:48.526+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:48.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:48.527+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:48.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:48.555+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:48.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:48.555+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:48.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:48.555+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:48.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:48.556+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:48.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:48.712+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:48.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:48.712+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:48.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:48.712+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:48.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:48.712+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:49.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:49.556+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:49.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:49.556+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:49.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:49.556+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:49.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:49.556+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:49.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:49.521+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:49.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:49.521+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:49.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:49.521+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:49.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:49.521+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:49.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:49.666+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:49.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:49.666+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:49.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:49.666+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:49.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:49.666+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:50.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:50.597+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:50.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:50.597+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:50.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:50.597+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:50.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:50.597+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:50.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:50.545+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:50.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:50.545+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:50.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:50.545+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:50.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:50.545+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:50.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:50.616+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:50.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:50.616+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:50.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:50.616+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:50.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:50.617+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:51.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:51.571+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:51.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:51.571+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:51.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:51.571+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:51.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:51.571+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:51.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:51.580+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:51.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:51.580+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:51.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:51.580+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:51.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:51.580+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:51.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:51.569+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:51.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:51.569+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:51.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:51.569+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:51.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:51.569+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:52.732 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:00:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:00:52.619+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 6528 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:00:52.732 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:52.595+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:52.732 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:52.595+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:52.732 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:52.595+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:52.732 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:52.595+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:52.733 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:52.526+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:52.733 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:52.526+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:52.733 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:52.526+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:52.733 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:52.526+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:52.733 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:52.553+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:52.733 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:52.553+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:52.733 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:52.553+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:52.733 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:52.553+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:53.477 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:53.491+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:53.477 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:53.491+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:53.477 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:53.491+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:53.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:53.491+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:53.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:53.565+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:53.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:53.565+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:53.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:53.565+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:53.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:53.565+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:53.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:53.574+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:53.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:53.574+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:53.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:53.574+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:53.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:53.574+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:54.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:54.469+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:54.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:54.469+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:54.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:54.469+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:54.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:54.470+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:54.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:54.592+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:54.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:54.592+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:54.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:54.592+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:54.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:54.593+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:54.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:54.581+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:54.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:54.581+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:54.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:54.581+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:54.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:54.581+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:55.553 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:55.504+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:55.554 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:55.504+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:55.554 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:55.504+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:55.554 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:55.504+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:55.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:55.610+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:55.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:55.610+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:55.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:55.610+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:55.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:55.611+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:55.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:55.568+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:55.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:55.568+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:55.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:55.568+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:55.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:55.568+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:56.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:56.572+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:56.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:56.572+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:56.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:56.572+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:56.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:56.573+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:56.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:56.539+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:56.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:56.539+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:56.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:56.539+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:56.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:56.539+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:56.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:56.536+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:56.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:56.536+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:56.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:56.536+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:56.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:56.536+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:57.734 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:00:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:00:57.621+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 6624 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:00:57.734 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:57.558+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:57.734 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:57.558+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:57.734 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:57.558+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:57.734 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:57.558+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:57.735 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:57.577+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:57.735 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:57.577+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:57.735 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:57.577+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:57.735 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:57.577+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:57.735 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:57.498+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:57.735 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:57.498+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:57.735 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:57.498+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:57.736 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:57.498+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:58.530 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:58.476+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:58.530 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:58.476+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:58.530 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:58.476+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:58.530 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:58.477+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:58.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:58.597+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:58.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:58.597+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:58.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:58.597+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:58.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:58.597+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:58.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:58.544+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:58.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:58.544+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:58.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:58.544+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:58.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:58.544+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:59.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:59.632+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:59.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:59.632+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:59.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:59.632+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:00:59.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:00:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:00:59.632+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:00:59.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:59.516+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:59.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:59.516+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:59.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:59.516+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:00:59.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:00:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:00:59.516+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:00:59.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:59.544+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:59.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:59.544+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:59.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:59.544+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:00:59.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:00:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:00:59.544+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:00.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:00.660+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:00.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:00.660+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:00.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:00.660+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:00.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:00.661+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:00.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:00.484+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:00.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:00.484+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:00.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:00.484+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:00.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:00.485+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:00.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:00.588+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:00.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:00.588+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:00.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:00.588+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:00.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:00.588+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:01.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:01.467+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:01.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:01.467+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:01.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:01.467+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:01.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:01.467+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:01.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:01.563+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:01.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:01.563+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:01.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:01.563+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:01.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:01.563+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:01.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:01.626+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:01.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:01.626+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:01.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:01.626+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:01.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:01.626+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:02.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:01:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:01:02.623+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 6720 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:01:02.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:02.418+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:02.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:02.418+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:02.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:02.418+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:02.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:02.418+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:02.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:02.620+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:02.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:02.620+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:02.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:02.620+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:02.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:02.621+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:02.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:02.577+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:02.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:02.577+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:02.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:02.577+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:02.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:02.577+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:03.625 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:03.449+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:03.626 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:03.449+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:03.626 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:03.449+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:03.626 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:03.449+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:03.626 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:03.543+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:03.626 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:03.543+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:03.626 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:03.543+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:03.626 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:03.544+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:03.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:03.640+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:03.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:03.640+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:03.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:03.640+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:03.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:03.640+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:04.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:04.606+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:04.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:04.606+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:04.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:04.606+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:04.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:04.606+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:04.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:04.543+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:04.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:04.543+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:04.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:04.543+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:04.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:04.543+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:04.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:04.482+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:04.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:04.482+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:04.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:04.482+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:04.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:04.482+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:05.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:05.654+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:05.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:05.654+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:05.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:05.654+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:05.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:05.655+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:05.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:05.564+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:05.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:05.564+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:05.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:05.564+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:05.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:05.564+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:05.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:05.477+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:05.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:05.477+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:05.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:05.477+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:05.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:05.477+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:06.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:06.454+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:06.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:06.454+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:06.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:06.454+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:06.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:06.455+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:06.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:06.695+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:06.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:06.695+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:06.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:06.695+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:06.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:06.695+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:06.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:06.533+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:06.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:06.533+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:06.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:06.533+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:06.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:06.533+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:07.735 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:01:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:01:07.625+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 6816 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:01:07.736 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:07.723+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:07.736 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:07.723+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:07.736 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:07.723+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:07.736 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:07.723+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:07.736 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:07.478+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:07.736 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:07.479+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:07.736 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:07.479+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:07.736 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:07.479+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:07.737 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:07.504+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:07.737 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:07.504+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:07.737 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:07.504+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:07.737 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:07.504+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:08.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:08.525+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:08.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:08.525+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:08.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:08.525+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:08.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:08.526+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:08.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:08.528+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:08.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:08.528+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:08.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:08.528+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:08.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:08.528+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:09.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:08.741+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:09.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:08.741+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:09.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:08.741+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:09.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:08.742+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:09.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:09.558+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:09.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:09.558+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:09.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:09.558+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:09.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:09.558+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:09.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:09.724+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:09.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:09.724+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:09.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:09.724+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:09.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:09.725+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:09.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:09.567+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:09.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:09.567+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:09.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:09.567+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:09.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:09.567+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:10.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:10.550+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:10.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:10.550+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:10.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:10.550+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:10.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:10.551+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:10.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:10.553+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:10.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:10.553+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:10.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:10.553+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:10.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:10.553+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:10.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:10.741+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:10.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:10.741+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:10.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:10.741+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:10.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:10.741+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:11.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:11.553+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:11.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:11.553+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:11.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:11.553+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:11.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:11.553+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:11.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:11.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:11.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:11.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:11.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:11.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:11.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:11.568+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:11.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:11.722+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:11.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:11.722+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:11.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:11.722+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:11.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:11.722+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:12.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:01:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:01:12.626+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 6911 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:01:12.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:12.563+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:12.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:12.563+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:12.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:12.563+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:12.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:12.563+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:12.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:12.723+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:12.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:12.723+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:12.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:12.723+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:12.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:12.723+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:12.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:12.598+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:12.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:12.598+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:12.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:12.598+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:12.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:12.598+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:13.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:13.564+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:13.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:13.564+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:13.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:13.564+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:13.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:13.564+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:13.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:13.531+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:13.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:13.531+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:13.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:13.531+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:13.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:13.531+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:14.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:13.698+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:14.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:13.698+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:14.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:13.698+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:14.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:13.699+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:14.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:14.537+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:14.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:14.537+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:14.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:14.537+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:14.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:14.537+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:14.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:14.576+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:14.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:14.576+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:14.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:14.576+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:14.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:14.576+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:14.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:14.730+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:14.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:14.730+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:14.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:14.730+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:14.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:14.730+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:15.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:15.544+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:15.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:15.544+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:15.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:15.544+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:15.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:15.544+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:15.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:15.540+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:15.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:15.540+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:15.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:15.540+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:15.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:15.540+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:15.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:15.702+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:15.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:15.702+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:15.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:15.702+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:15.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:15.702+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:16.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:16.571+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:16.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:16.571+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:16.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:16.571+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:16.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:16.571+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:16.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:16.589+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:16.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:16.589+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:16.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:16.589+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:16.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:16.589+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:16.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:16.686+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:16.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:16.686+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:16.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:16.686+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:16.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:16.686+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:17.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:01:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:01:17.628+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 7008 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:01:17.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:17.583+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:17.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:17.583+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:17.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:17.583+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:17.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:17.583+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:17.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:17.597+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:17.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:17.597+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:17.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:17.597+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:17.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:17.597+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:17.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:17.662+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:17.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:17.662+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:17.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:17.662+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:17.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:17.662+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:18.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:18.550+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:18.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:18.550+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:18.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:18.550+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:18.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:18.550+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:18.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:18.596+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:18.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:18.596+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:18.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:18.596+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:18.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:18.597+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:18.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:18.671+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:18.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:18.671+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:18.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:18.671+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:18.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:18.671+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:19.846 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:19.636+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:19.847 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:19.636+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:19.847 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:19.636+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:19.847 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:19.637+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:19.847 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:19.675+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:19.847 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:19.675+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:19.847 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:19.675+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:19.847 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:19.675+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:19.848 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:19.510+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:19.848 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:19.510+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:19.848 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:19.510+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:19.848 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:19.510+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:20.602 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T03:01:20.615+0000 7f415c480700 0 monclient(hunting): authenticate timed out after 300 2024-04-06T03:01:20.602 INFO:teuthology.orchestra.run.smithi105.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-04-06T03:01:20.914 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:20.670+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:20.914 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:20.670+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:20.914 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:20.670+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:20.914 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:20.670+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:20.915 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:20.543+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:20.915 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:20.543+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:20.915 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:20.543+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:20.915 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:20.543+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:20.917 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:20.665+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:20.917 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:20.665+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:20.917 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:20.665+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:20.918 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:20.665+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:21.367 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-04-06T03:01:21.585 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/mon.smithi105/config 2024-04-06T03:01:21.605 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:21.545+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:21.605 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:21.545+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:21.605 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:21.545+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:21.605 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:21.545+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:21.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:21.619+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:21.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:21.619+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:21.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:21.619+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:21.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:21.619+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:21.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:21.700+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:21.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:21.700+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:21.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:21.700+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:21.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:21.701+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:22.659 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:01:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:01:22.629+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 7104 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:01:22.659 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:22.640+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:22.659 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:22.640+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:22.659 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:22.640+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:22.659 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:22.641+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:22.659 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:22.595+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:22.659 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:22.595+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:22.659 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:22.595+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:22.660 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:22.595+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:22.919 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:22.673+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:22.919 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:22.673+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:22.919 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:22.673+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:22.919 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:22.673+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:23.646+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:23.646+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:23.646+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:23.646+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:23.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:23.612+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:23.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:23.612+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:23.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:23.612+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:23.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:23.612+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:23.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:23.591+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:23.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:23.591+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:23.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:23.591+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:23.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:23.592+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:24.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:24.647+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:24.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:24.647+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:24.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:24.647+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:24.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:24.648+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:24.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:24.596+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:24.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:24.596+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:24.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:24.596+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:24.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:24.596+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:24.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:24.641+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:24.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:24.641+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:24.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:24.641+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:24.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:24.641+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:25.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:25.614+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.032822+0000 front 2024-04-06T02:50:58.032830+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:25.963 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:25.614+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.032811+0000 front 2024-04-06T02:50:58.032857+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:25.963 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:25.614+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.032843+0000 front 2024-04-06T02:50:58.032878+0000 (oldest deadline 2024-04-06T02:51:22.132838+0000) 2024-04-06T03:01:25.963 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:25.614+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:25.964 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:25.642+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:25.964 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:25.643+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:25.964 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:25.643+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:25.964 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:25.643+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:25.964 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:25.644+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:25.964 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:25.644+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:25.964 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:25.644+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:25.964 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:25.644+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:26.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:26.645+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:26.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:26.620+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:26.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:26.620+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:26.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:26.620+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:26.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:26.621+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:26.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:26.667+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:26.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:26.667+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:26.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:26.667+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:26.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:26.667+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:27.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:01:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:01:27.631+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 7200 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:01:27.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:27.605+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:27.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:27.605+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:27.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:27.605+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:27.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:27.605+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:27.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:27.605+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:27.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:27.622+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:27.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:27.622+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:27.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:27.622+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:27.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:27.623+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:28.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:28.572+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:28.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:28.590+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:58.534864+0000 front 2024-04-06T02:50:58.534822+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:28.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:28.590+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:58.534775+0000 front 2024-04-06T02:50:58.534805+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:28.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:28.590+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:58.534850+0000 front 2024-04-06T02:50:58.534834+0000 (oldest deadline 2024-04-06T02:51:20.834802+0000) 2024-04-06T03:01:28.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:28.590+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:28.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:28.636+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:28.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:28.636+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:28.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:28.636+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:28.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:28.636+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:29.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:29.605+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:29.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:29.557+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:29.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:29.613+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:29.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:29.613+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:29.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:29.613+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:29.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:29.614+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:30.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:30.590+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:30.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:30.534+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:30.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:30.597+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:30.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:30.597+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:30.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:30.597+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:30.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:30.597+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:31.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:31.617+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:31.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:31.551+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:31.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:31.636+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:31.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:31.636+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:31.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:31.636+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:31.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:31.637+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:32.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:32.574+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:32.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:32.538+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:32.926 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:01:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:01:32.632+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 7295 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:01:32.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:32.644+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:32.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:32.644+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:32.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:32.644+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:32.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:32.644+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:33.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:33.615+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:33.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:33.494+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:33.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:33.658+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:33.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:33.658+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:33.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:33.658+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:33.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:33.659+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:34.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:34.637+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:34.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:34.468+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:34.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:34.633+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:34.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:34.633+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:34.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:34.633+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:34.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:34.634+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:35.590 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:35.438+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:35.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:35.660+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:35.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:35.604+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 since back 2024-04-06T02:50:59.210937+0000 front 2024-04-06T02:50:59.211067+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:35.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:35.604+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 since back 2024-04-06T02:50:59.211027+0000 front 2024-04-06T02:50:59.210972+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:35.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:35.604+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 since back 2024-04-06T02:50:59.210926+0000 front 2024-04-06T02:50:59.211056+0000 (oldest deadline 2024-04-06T02:51:23.910848+0000) 2024-04-06T03:01:35.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:35.604+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:36.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:36.630+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:36.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:36.402+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:37.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:36.698+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:37.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:37.426+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:37.676 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:01:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:01:37.634+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 7391 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:01:37.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:37.596+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:38.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:37.706+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:38.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:38.598+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:38.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:38.691+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:38.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:38.474+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:39.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:39.603+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:39.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:39.687+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:39.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:39.438+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:40.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:40.617+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:40.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:40.717+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:40.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:40.472+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:41.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:41.604+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:41.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:41.490+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:42.000 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:41.750+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:42.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:42.520+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:42.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:01:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:01:42.635+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 7486 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:01:42.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:42.556+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:42.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:42.757+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:43.727 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:43.584+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:43.727 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:43.484+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:44.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:43.757+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:44.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:44.602+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:44.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:44.450+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:45.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:44.799+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:45.834 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:45.651+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:45.835 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:45.447+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:46.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:45.848+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:46.845 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:46.638+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:46.845 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:46.466+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:47.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:46.859+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:47.750 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:01:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:01:47.637+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 7581 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:01:47.750 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:47.440+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:47.751 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:47.649+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:48.178 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:47.809+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:48.178 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:47.809+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:48.178 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:47.809+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:48.178 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:47.810+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:48.759 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:48.681+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:48.759 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:48.445+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:49.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:48.773+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:49.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:48.773+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:49.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:48.773+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:49.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:48.773+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:49.801 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:49.645+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:49.802 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:49.453+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:50.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:49.815+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:50.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:49.815+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:50.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:49.816+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:50.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:49.816+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:50.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:50.660+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:50.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:50.432+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:51.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:50.834+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:51.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:50.834+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:51.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:50.834+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:51.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:50.835+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:51.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:51.611+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:51.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:51.404+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:52.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:51.794+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:52.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:51.794+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:52.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:51.794+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:52.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:51.794+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:52.624 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:52.361+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:52.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:52.640+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:52.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:01:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:01:52.638+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 7675 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:01:52.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:52.812+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:52.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:52.812+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:52.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:52.812+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:52.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:52.813+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:53.446 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:53.327+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:01:53.446 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:53.327+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:01:53.446 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:53.327+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:01:53.446 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:53.327+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:53.794 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:53.644+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:54.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:53.808+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:54.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:53.808+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:54.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:53.808+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:54.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:53.809+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:54.634 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:54.305+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:01:54.634 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:54.305+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:01:54.634 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:54.305+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:01:54.634 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:54.305+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:54.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:54.648+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:54.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:54.838+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:54.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:54.838+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:54.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:54.838+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:54.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:54.838+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:55.619 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:55.321+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:01:55.619 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:55.321+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:01:55.619 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:55.321+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:01:55.619 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:55.321+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:55.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:55.633+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:55.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:55.836+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:55.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:55.837+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:55.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:55.837+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:55.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:55.837+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:56.624 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:56.278+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:01:56.624 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:56.278+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:01:56.624 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:56.278+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:01:56.624 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:56.278+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:56.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:56.638+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:56.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:56.885+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:56.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:56.885+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:56.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:56.885+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:56.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:56.886+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:57.626 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:57.256+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:01:57.626 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:57.256+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:01:57.627 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:57.256+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:01:57.627 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:57.256+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:57.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:01:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:01:57.640+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 7770 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:01:57.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:57.668+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:57.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:57.890+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:57.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:57.890+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:57.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:57.890+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:57.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:57.890+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:58.663 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:58.280+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:01:58.663 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:58.280+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:01:58.664 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:58.280+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:01:58.664 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:58.281+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:58.916 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:58.678+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:01:58.917 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:58.678+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:01:58.917 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:58.678+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:01:58.917 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:58.678+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:01:59.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:58.931+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:59.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:58.931+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:59.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:58.931+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:01:59.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:58.931+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:01:59.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:59.281+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:01:59.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:59.281+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:01:59.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:59.281+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:01:59.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:01:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:01:59.281+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:00.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:59.695+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:00.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:59.695+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:00.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:59.695+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:00.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:01:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:01:59.696+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:00.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:59.889+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:00.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:59.889+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:00.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:59.889+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:00.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:01:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:01:59.889+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:00.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:00.243+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:00.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:00.243+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:00.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:00.243+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:00.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:00.243+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:01.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:00.694+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:01.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:00.694+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:01.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:00.694+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:01.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:00.695+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:01.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:00.878+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:01.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:00.878+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:01.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:00.878+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:01.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:00.878+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:01.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:01.279+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:01.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:01.279+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:01.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:01.279+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:01.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:01.279+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:02.000 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:01.870+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:02.000 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:01.870+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:02.000 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:01.870+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:02.000 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:01.871+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:02.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:01.743+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:02.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:01.743+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:02.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:01.743+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:02.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:01.744+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:02.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:02.259+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:02.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:02.259+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:02.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:02.259+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:02.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:02.259+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:02.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:02:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:02:02.642+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 7864 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:02:02.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:02.706+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:02.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:02.706+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:02.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:02.706+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:02.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:02.706+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:02.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:02.828+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:02.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:02.828+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:02.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:02.828+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:02.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:02.829+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:03.547 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:03.295+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:03.548 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:03.295+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:03.548 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:03.295+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:03.548 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:03.295+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:03.861 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:03.682+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:03.861 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:03.682+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:03.861 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:03.682+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:03.861 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:03.682+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:04.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:03.875+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:04.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:03.875+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:04.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:03.875+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:04.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:03.876+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:04.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:04.339+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:04.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:04.339+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:04.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:04.339+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:04.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:04.339+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:05.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:04.868+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:05.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:04.868+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:05.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:04.868+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:05.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:04.868+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:05.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:04.727+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:05.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:04.727+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:05.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:04.727+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:05.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:04.728+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:05.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:05.332+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:05.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:05.332+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:05.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:05.332+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:05.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:05.333+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:06.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:05.838+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:06.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:05.838+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:06.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:05.838+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:06.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:05.839+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:06.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:05.757+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:06.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:05.757+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:06.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:05.757+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:06.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:05.757+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:06.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:06.318+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:06.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:06.318+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:06.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:06.318+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:06.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:06.318+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:07.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:06.725+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:07.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:06.725+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:07.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:06.725+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:07.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:06.726+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:07.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:06.857+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:07.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:06.857+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:07.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:06.857+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:07.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:06.857+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:07.629 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:07.343+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:07.630 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:07.343+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:07.630 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:07.343+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:07.630 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:07.343+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:07.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:02:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:02:07.643+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 7958 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:02:07.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:07.773+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:07.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:07.773+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:07.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:07.773+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:07.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:07.774+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:07.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:07.809+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:07.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:07.809+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:07.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:07.809+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:07.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:07.810+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:08.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:08.381+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:08.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:08.381+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:08.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:08.381+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:08.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:08.381+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:09.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:08.807+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:09.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:08.807+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:09.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:08.807+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:09.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:08.808+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:09.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:08.799+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:09.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:08.799+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:09.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:08.799+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:09.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:08.799+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:09.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:09.335+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:09.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:09.335+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:09.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:09.335+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:09.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:09.335+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:10.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:09.758+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:10.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:09.758+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:10.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:09.758+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:10.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:09.759+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:10.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:09.768+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:10.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:09.768+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:10.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:09.768+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:10.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:09.769+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:10.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:10.371+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:10.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:10.371+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:10.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:10.371+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:10.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:10.371+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:11.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:10.796+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:11.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:10.796+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:11.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:10.796+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:11.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:10.796+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:11.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:10.793+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:11.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:10.793+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:11.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:10.793+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:11.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:10.793+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:11.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:11.338+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:11.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:11.338+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:11.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:11.338+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:11.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:11.338+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:12.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:11.770+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:12.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:11.770+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:12.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:11.770+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:12.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:11.771+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:12.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:11.791+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:12.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:11.791+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:12.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:11.791+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:12.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:11.791+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:12.631 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:12.305+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:12.631 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:12.305+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:12.632 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:12.305+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:12.632 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:12.305+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:12.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:02:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:02:12.645+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 8051 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:02:12.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:12.769+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:12.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:12.769+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:12.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:12.769+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:12.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:12.769+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:12.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:12.748+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:12.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:12.748+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:12.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:12.748+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:12.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:12.748+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:13.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:13.333+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:13.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:13.333+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:13.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:13.333+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:13.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:13.334+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:14.023 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:13.747+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:14.023 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:13.747+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:14.023 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:13.747+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:14.023 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:13.747+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:14.023 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:13.815+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:14.023 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:13.815+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:14.023 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:13.815+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:14.023 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:13.815+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:14.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:14.352+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:14.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:14.352+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:14.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:14.352+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:14.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:14.352+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:15.142 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:14.793+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:15.142 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:14.793+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:15.142 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:14.793+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:15.142 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:14.793+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:15.142 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:14.753+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:15.143 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:14.753+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:15.143 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:14.753+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:15.143 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:14.753+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:15.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:15.394+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:15.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:15.394+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:15.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:15.394+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:15.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:15.394+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:16.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:15.828+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:16.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:15.828+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:16.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:15.828+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:16.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:15.828+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:16.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:15.751+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:16.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:15.751+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:16.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:15.751+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:16.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:15.751+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:16.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:16.436+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:16.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:16.436+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:16.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:16.436+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:16.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:16.436+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:17.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:16.763+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:17.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:16.763+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:17.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:16.763+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:17.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:16.763+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:17.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:16.788+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:17.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:16.788+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:17.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:16.788+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:17.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:16.788+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:17.646 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:02:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:02:17.647+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 8147 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:02:17.646 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:17.407+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:17.646 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:17.407+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:17.646 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:17.407+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:17.646 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:17.407+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:17.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:17.781+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:17.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:17.781+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:17.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:17.781+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:17.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:17.781+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:17.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:17.766+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:17.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:17.766+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:17.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:17.766+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:17.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:17.766+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:18.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:18.370+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:18.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:18.370+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:18.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:18.370+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:18.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:18.371+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:19.073 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:18.780+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:19.073 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:18.780+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:19.073 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:18.780+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:19.073 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:18.780+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:19.073 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:18.732+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:19.074 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:18.732+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:19.074 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:18.732+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:19.074 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:18.732+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:19.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:19.327+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:19.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:19.327+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:19.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:19.327+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:19.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:19.327+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:19.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:19.685+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:19.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:19.685+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:19.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:19.685+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:19.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:19.685+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:19.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:19.781+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:19.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:19.781+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:19.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:19.781+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:19.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:19.781+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:20.631 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:20.295+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:20.631 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:20.295+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:20.631 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:20.295+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:20.631 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:20.295+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:20.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:20.754+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:20.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:20.754+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:20.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:20.754+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:20.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:20.754+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:20.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:20.646+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:20.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:20.646+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:20.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:20.646+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:20.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:20.646+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:21.626 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:21.264+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:21.626 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:21.264+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:21.626 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:21.264+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:21.626 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:21.264+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:21.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:21.765+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:21.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:21.765+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:21.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:21.765+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:21.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:21.766+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:21.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:21.641+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:21.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:21.641+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:21.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:21.641+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:21.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:21.641+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:22.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:22.215+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:22.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:22.215+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:22.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:22.215+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:22.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:22.215+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:22.737 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:02:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:02:22.648+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 8241 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:02:22.738 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:22.601+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:22.738 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:22.601+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:22.738 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:22.601+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:22.738 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:22.601+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:23.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:22.752+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:23.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:22.752+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:23.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:22.752+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:23.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:22.752+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:23.538 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:23.248+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:23.538 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:23.248+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:23.538 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:23.248+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:23.538 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:23.248+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:23.552+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:23.552+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:23.552+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:23.552+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:23.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:23.792+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:23.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:23.792+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:23.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:23.792+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:23.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:23.792+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:24.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:24.265+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:24.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:24.265+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:24.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:24.265+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:24.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:24.265+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:24.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:24.753+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:24.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:24.753+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:24.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:24.753+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:24.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:24.753+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:24.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:24.534+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:24.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:24.534+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:24.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:24.534+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:24.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:24.534+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:25.568 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:25.282+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:25.568 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:25.282+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:25.568 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:25.282+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:25.569 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:25.282+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:25.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:25.766+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:25.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:25.766+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:25.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:25.766+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:25.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:25.766+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:25.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:25.583+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:25.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:25.583+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:25.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:25.583+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:25.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:25.583+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:26.571 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:26.300+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:26.571 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:26.300+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:26.571 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:26.300+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:26.571 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:26.300+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:26.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:26.738+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:26.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:26.738+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:26.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:26.738+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:26.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:26.738+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:26.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:26.585+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:26.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:26.585+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:26.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:26.585+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:26.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:26.585+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:27.636 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:27.559+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:27.636 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:27.559+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:27.636 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:27.559+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:27.636 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:27.559+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:27.636 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:27.338+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:27.637 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:27.338+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:27.637 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:27.338+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:27.637 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:27.338+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:27.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:02:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:02:27.649+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 8335 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:02:27.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:27.768+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:27.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:27.768+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:27.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:27.768+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:27.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:27.768+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:28.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:28.594+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:28.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:28.594+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:28.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:28.594+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:28.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:28.594+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:28.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:28.373+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:28.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:28.373+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:28.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:28.373+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:28.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:28.373+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:29.172 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:28.729+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:29.172 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:28.729+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:29.172 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:28.729+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:29.172 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:28.729+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:29.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:29.417+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:29.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:29.417+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:29.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:29.417+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:29.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:29.417+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:29.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:29.697+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:29.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:29.697+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:29.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:29.697+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:29.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:29.697+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:29.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:29.605+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:29.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:29.605+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:29.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:29.605+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:29.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:29.605+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:30.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:30.569+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:30.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:30.569+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:30.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:30.569+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:30.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:30.569+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:30.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:30.436+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:30.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:30.436+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:30.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:30.436+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:30.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:30.436+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:31.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:30.716+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:31.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:30.716+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:31.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:30.716+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:31.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:30.716+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:31.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:31.599+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:31.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:31.599+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:31.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:31.599+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:31.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:31.599+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:31.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:31.420+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:31.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:31.420+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:31.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:31.420+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:31.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:31.421+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:32.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:31.693+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:32.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:31.693+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:32.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:31.693+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:32.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:31.694+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:32.623 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:32.380+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:32.623 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:32.380+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:32.623 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:32.380+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:32.623 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:32.380+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:32.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:02:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:02:32.652+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 8428 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:02:32.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:32.710+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:32.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:32.710+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:32.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:32.710+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:32.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:32.711+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:32.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:32.637+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:32.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:32.637+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:32.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:32.637+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:32.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:32.638+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:33.574 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:33.333+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:33.574 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:33.333+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:33.574 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:33.333+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:33.574 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:33.333+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:33.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:33.730+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:33.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:33.730+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:33.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:33.730+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:33.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:33.731+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:33.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:33.681+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:33.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:33.681+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:33.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:33.681+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:33.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:33.681+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:34.641 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:34.366+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:34.642 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:34.366+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:34.642 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:34.366+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:34.642 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:34.366+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:34.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:34.723+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:34.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:34.723+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:34.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:34.723+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:34.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:34.724+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:34.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:34.656+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:34.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:34.656+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:34.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:34.656+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:34.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:34.656+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:35.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:35.357+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:35.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:35.357+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:35.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:35.357+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:35.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:35.357+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:36.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:35.712+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:36.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:35.712+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:36.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:35.712+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:36.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:35.712+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:36.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:35.691+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:36.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:35.691+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:36.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:35.691+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:36.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:35.691+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:36.630 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:36.382+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:36.630 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:36.382+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:36.630 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:36.382+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:36.630 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:36.382+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:36.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:36.736+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:36.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:36.736+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:36.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:36.736+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:36.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:36.736+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:36.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:36.644+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:36.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:36.644+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:36.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:36.644+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:36.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:36.645+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:37.639 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:37.382+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:37.712 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:37.382+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:37.713 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:37.382+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:37.713 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:37.382+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:37.713 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:37.605+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:37.713 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:37.605+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:37.713 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:37.605+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:37.713 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:37.606+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:37.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:02:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:02:37.653+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 8523 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:02:37.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:37.707+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:37.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:37.707+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:37.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:37.707+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:37.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:37.707+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:38.653 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:38.402+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:38.653 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:38.402+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:38.653 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:38.402+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:38.654 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:38.403+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:38.654 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:38.605+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:38.654 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:38.605+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:38.654 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:38.605+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:38.654 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:38.605+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:38.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:38.667+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:38.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:38.667+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:38.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:38.667+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:38.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:38.668+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:39.618 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:39.448+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:39.618 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:39.448+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:39.619 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:39.448+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:39.619 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:39.448+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:39.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:39.679+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:39.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:39.679+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:39.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:39.679+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:39.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:39.680+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:39.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:39.633+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:39.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:39.633+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:39.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:39.633+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:39.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:39.633+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:40.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:40.689+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:40.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:40.690+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:40.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:40.690+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:40.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:40.690+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:40.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:40.607+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:40.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:40.607+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:40.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:40.607+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:40.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:40.607+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:40.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:40.442+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:40.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:40.442+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:40.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:40.442+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:40.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:40.442+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:41.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:41.562+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:41.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:41.562+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:41.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:41.562+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:41.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:41.562+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:41.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:41.401+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:41.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:41.401+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:41.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:41.401+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:41.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:41.401+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:42.000 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:41.732+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:42.000 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:41.732+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:42.000 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:41.732+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:42.000 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:41.732+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:42.641 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:42.593+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:42.641 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:42.593+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:42.641 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:42.593+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:42.641 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:42.594+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:42.641 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:42.402+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:42.642 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:42.402+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:42.642 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:42.402+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:42.642 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:42.402+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:42.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:02:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:02:42.655+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 8617 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:02:42.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:42.781+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:42.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:42.781+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:42.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:42.781+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:42.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:42.782+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:43.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:43.636+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:43.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:43.636+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:43.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:43.636+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:43.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:43.636+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:43.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:43.423+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:43.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:43.423+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:43.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:43.423+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:43.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:43.423+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:44.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:43.779+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:44.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:43.779+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:44.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:43.779+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:44.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:43.779+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:44.607 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:44.397+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:44.607 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:44.397+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:44.607 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:44.397+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:44.607 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:44.397+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:44.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:44.804+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:44.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:44.804+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:44.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:44.804+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:44.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:44.804+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:44.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:44.621+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:44.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:44.621+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:44.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:44.621+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:44.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:44.622+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:45.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:45.619+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:45.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:45.619+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:45.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:45.619+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:45.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:45.619+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:45.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:45.384+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:45.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:45.384+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:45.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:45.384+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:45.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:45.384+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:46.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:45.766+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:46.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:45.766+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:46.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:45.766+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:46.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:45.767+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:46.633 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:46.397+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:46.634 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:46.397+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:46.634 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:46.397+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:46.634 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:46.398+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:46.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:46.793+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:46.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:46.793+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:46.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:46.793+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:46.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:46.793+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:46.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:46.648+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:46.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:46.648+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:46.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:46.648+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:46.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:46.648+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:47.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:02:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:02:47.657+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 8711 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:02:47.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:47.603+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:47.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:47.603+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:47.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:47.603+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:47.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:47.603+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:47.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:47.428+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:47.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:47.428+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:47.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:47.428+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:47.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:47.428+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:48.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:47.788+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:48.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:47.788+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:48.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:47.788+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:48.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:47.788+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:48.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:48.631+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:48.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:48.631+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:48.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:48.631+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:48.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:48.631+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:48.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:48.409+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:48.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:48.409+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:48.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:48.409+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:48.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:48.409+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:49.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:48.796+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:49.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:48.797+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:49.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:48.797+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:49.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:48.797+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:49.649 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:49.420+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:49.649 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:49.420+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:49.649 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:49.420+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:49.650 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:49.420+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:49.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:49.810+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:49.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:49.810+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:49.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:49.810+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:49.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:49.811+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:49.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:49.664+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:49.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:49.664+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:49.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:49.664+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:49.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:49.664+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:50.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:50.405+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:50.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:50.405+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:50.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:50.405+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:50.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:50.405+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:51.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:50.849+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:51.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:50.849+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:51.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:50.849+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:51.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:50.850+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:51.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:50.691+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:51.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:50.691+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:51.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:50.691+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:51.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:50.691+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:51.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:51.672+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:51.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:51.672+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:51.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:51.672+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:51.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:51.672+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:51.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:51.426+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:51.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:51.426+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:51.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:51.426+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:51.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:51.426+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:52.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:51.861+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:52.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:51.861+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:52.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:51.861+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:52.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:51.862+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:52.771 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:02:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:02:52.658+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 8804 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:02:52.771 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:52.688+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:52.771 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:52.688+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:52.771 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:52.688+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:52.771 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:52.688+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:52.771 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:52.443+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:52.772 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:52.443+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:52.772 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:52.443+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:52.772 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:52.443+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:53.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:52.849+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:53.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:52.850+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:53.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:52.850+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:53.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:52.850+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:53.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:53.665+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:53.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:53.665+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:53.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:53.665+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:53.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:53.665+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:53.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:53.421+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:53.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:53.421+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:53.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:53.421+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:53.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:53.421+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:54.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:53.868+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:54.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:53.868+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:54.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:53.868+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:54.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:53.868+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:54.673 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:54.413+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:54.673 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:54.413+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:54.673 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:54.413+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:54.673 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:54.413+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:54.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:54.840+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:54.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:54.840+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:54.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:54.840+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:54.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:54.840+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:54.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:54.687+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:54.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:54.687+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:54.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:54.687+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:54.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:54.687+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:55.847 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:55.441+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:55.847 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:55.441+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:55.847 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:55.441+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:55.848 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:55.441+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:55.848 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:55.662+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:55.848 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:55.662+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:55.848 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:55.662+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:55.848 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:55.662+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:56.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:55.862+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:56.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:55.862+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:56.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:55.862+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:56.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:55.862+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:56.841 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:56.634+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:56.841 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:56.634+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:56.841 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:56.634+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:56.841 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:56.634+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:56.841 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:56.484+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:56.842 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:56.484+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:56.842 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:56.484+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:56.842 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:56.484+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:57.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:56.855+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:57.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:56.855+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:57.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:56.855+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:57.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:56.855+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:57.772 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:02:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:02:57.660+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 8898 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:02:57.772 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:57.656+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:57.772 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:57.656+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:57.772 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:57.656+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:57.772 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:57.657+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:57.772 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:57.532+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:57.773 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:57.532+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:57.773 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:57.532+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:57.773 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:57.532+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:58.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:57.891+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:58.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:57.891+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:58.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:57.891+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:58.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:57.891+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:58.905 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:58.543+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:58.905 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:58.543+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:58.905 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:58.543+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:58.905 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:58.543+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:58.905 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:58.654+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:58.905 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:58.654+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:58.905 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:58.654+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:58.906 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:58.654+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:02:59.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:58.919+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:59.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:58.919+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:59.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:58.919+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:02:59.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:58.919+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:59.910 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:59.554+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:59.911 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:59.554+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:59.911 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:59.554+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:02:59.911 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:02:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:02:59.554+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:02:59.911 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:59.633+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:59.911 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:59.633+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:59.911 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:59.633+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:02:59.911 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:02:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:02:59.633+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:00.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:59.925+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:00.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:59.925+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:00.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:59.925+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:00.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:02:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:02:59.925+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:00.870 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:00.592+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:00.870 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:00.592+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:00.870 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:00.592+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:00.870 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:00.592+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:00.871 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:00.507+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:00.871 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:00.507+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:00.871 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:00.507+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:00.871 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:00.507+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:01.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:00.884+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:01.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:00.884+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:01.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:00.884+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:01.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:00.884+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:01.864 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:01.628+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:01.864 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:01.628+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:01.864 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:01.628+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:01.865 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:01.628+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:01.865 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:01.553+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:01.865 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:01.553+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:01.865 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:01.553+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:01.865 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:01.553+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:02.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:01.879+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:02.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:01.879+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:02.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:01.879+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:02.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:01.879+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:02.773 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:03:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:03:02.662+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 8991 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:03:02.774 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:02.531+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:02.774 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:02.531+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:02.774 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:02.531+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:02.774 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:02.531+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:02.774 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:02.661+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:02.774 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:02.661+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:02.774 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:02.661+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:02.775 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:02.662+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:03.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:02.861+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:03.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:02.861+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:03.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:02.861+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:03.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:02.861+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:03.866 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:03.528+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:03.866 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:03.528+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:03.867 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:03.528+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:03.867 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:03.528+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:03.867 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:03.676+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:03.867 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:03.676+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:03.867 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:03.676+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:03.867 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:03.677+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:04.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:03.881+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:04.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:03.881+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:04.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:03.881+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:04.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:03.881+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:04.654 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:04.500+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:04.654 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:04.500+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:04.655 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:04.500+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:04.655 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:04.500+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:04.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:04.843+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:04.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:04.843+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:04.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:04.843+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:04.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:04.843+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:04.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:04.669+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:04.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:04.669+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:04.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:04.669+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:04.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:04.669+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:05.834 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:05.695+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:05.834 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:05.695+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:05.834 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:05.695+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:05.834 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:05.696+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:05.835 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:05.467+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:05.835 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:05.467+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:05.835 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:05.467+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:05.835 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:05.467+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:06.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:05.848+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:06.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:05.848+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:06.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:05.848+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:06.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:05.849+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:06.856 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:06.474+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:06.856 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:06.474+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:06.856 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:06.474+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:06.856 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:06.474+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:06.856 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:06.723+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:06.856 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:06.723+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:06.856 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:06.723+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:06.857 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:06.723+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:07.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:06.870+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:07.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:06.870+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:07.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:06.870+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:07.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:06.870+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:07.772 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:03:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:03:07.664+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 9084 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:03:07.772 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:07.682+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:07.772 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:07.682+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:07.773 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:07.682+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:07.773 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:07.683+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:07.773 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:07.506+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:07.773 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:07.506+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:07.773 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:07.506+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:07.773 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:07.506+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:08.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:07.841+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:08.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:07.841+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:08.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:07.841+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:08.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:07.841+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:08.813 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:08.640+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:08.813 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:08.640+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:08.813 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:08.640+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:08.813 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:08.640+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:08.813 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:08.473+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:08.813 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:08.473+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:08.813 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:08.473+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:08.814 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:08.473+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:09.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:08.827+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:09.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:08.827+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:09.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:08.827+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:09.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:08.827+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:09.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:09.610+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:09.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:09.610+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:09.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:09.610+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:09.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:09.611+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:09.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:09.425+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:09.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:09.425+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:09.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:09.425+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:09.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:09.425+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:10.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:09.855+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:10.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:09.855+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:10.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:09.855+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:10.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:09.855+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:10.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:10.634+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:10.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:10.634+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:10.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:10.634+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:10.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:10.634+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:10.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:10.433+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:10.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:10.433+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:10.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:10.433+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:10.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:10.434+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:11.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:10.872+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:11.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:10.872+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:11.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:10.872+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:11.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:10.872+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:11.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:11.630+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:11.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:11.630+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:11.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:11.630+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:11.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:11.630+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:11.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:11.419+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:11.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:11.419+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:11.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:11.419+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:11.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:11.419+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:12.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:11.898+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:12.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:11.898+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:12.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:11.898+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:12.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:11.898+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:12.675 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:03:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:03:12.665+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 9176 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:03:12.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:12.585+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:12.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:12.585+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:12.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:12.585+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:12.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:12.585+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:12.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:12.437+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:12.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:12.437+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:12.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:12.437+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:12.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:12.437+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:13.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:12.861+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:13.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:12.861+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:13.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:12.861+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:13.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:12.861+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:13.889 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:13.624+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:13.890 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:13.624+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:13.890 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:13.624+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:13.890 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:13.625+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:13.890 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:13.480+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:13.890 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:13.480+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:13.890 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:13.480+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:13.890 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:13.480+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:14.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:13.904+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:14.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:13.904+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:14.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:13.904+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:14.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:13.904+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:14.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:14.671+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:14.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:14.671+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:14.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:14.671+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:14.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:14.672+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:14.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:14.439+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:14.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:14.439+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:14.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:14.439+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:14.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:14.439+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:15.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:14.869+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:15.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:14.869+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:15.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:14.869+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:15.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:14.870+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:15.853 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:15.663+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:15.853 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:15.663+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:15.853 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:15.663+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:15.853 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:15.664+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:15.854 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:15.488+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:15.854 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:15.488+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:15.854 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:15.488+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:15.854 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:15.488+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:16.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:15.867+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:16.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:15.867+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:16.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:15.867+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:16.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:15.868+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:16.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:16.439+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:16.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:16.439+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:16.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:16.439+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:16.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:16.439+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:16.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:16.620+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:16.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:16.620+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:16.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:16.620+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:16.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:16.620+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:17.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:16.914+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:17.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:16.914+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:17.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:16.914+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:17.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:16.914+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:17.776 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:03:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:03:17.666+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 9272 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:03:17.776 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:17.638+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:17.777 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:17.638+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:17.777 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:17.638+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:17.777 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:17.638+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:17.777 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:17.465+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:17.777 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:17.465+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:17.777 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:17.465+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:17.777 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:17.465+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:18.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:17.903+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:18.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:17.903+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:18.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:17.903+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:18.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:17.903+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:18.851 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:18.658+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:18.851 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:18.658+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:18.851 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:18.658+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:18.851 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:18.658+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:18.852 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:18.472+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:18.852 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:18.472+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:18.852 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:18.472+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:18.852 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:18.472+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:19.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:18.865+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:19.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:18.865+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:19.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:18.865+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:19.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:18.865+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:19.855 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:19.639+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:19.855 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:19.639+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:19.855 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:19.639+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:19.855 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:19.639+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:19.855 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:19.467+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:19.855 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:19.467+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:19.855 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:19.467+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:19.856 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:19.467+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:20.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:19.869+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:20.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:19.869+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:20.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:19.869+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:20.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:19.869+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:20.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:20.629+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:20.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:20.629+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:20.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:20.629+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:20.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:20.630+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:20.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:20.438+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:20.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:20.438+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:20.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:20.438+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:20.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:20.438+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:21.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:20.860+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:21.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:20.860+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:21.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:20.860+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:21.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:20.861+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:21.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:21.606+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:21.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:21.606+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:21.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:21.606+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:21.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:21.606+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:21.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:21.434+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:21.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:21.434+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:21.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:21.434+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:21.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:21.435+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:22.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:21.907+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:22.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:21.907+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:22.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:21.907+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:22.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:21.908+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:22.623 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:22.432+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:22.623 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:22.432+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:22.623 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:22.432+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:22.624 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:22.433+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:22.914 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:03:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:03:22.669+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 9365 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:03:22.915 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:22.637+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:22.915 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:22.637+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:22.915 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:22.637+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:22.915 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:22.638+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:23.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:22.929+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:23.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:22.929+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:23.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:22.929+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:23.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:22.929+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:23.644 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:23.401+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:23.644 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:23.401+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:23.645 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:23.401+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:23.645 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:23.401+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:23.658+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:23.658+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:23.658+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:23.659+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:24.373 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:23.969+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:24.373 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:23.969+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:24.373 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:23.970+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:24.373 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:23.970+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:24.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:24.387+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:24.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:24.387+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:24.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:24.387+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:24.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:24.387+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:24.959 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:24.691+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:24.959 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:24.691+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:24.959 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:24.691+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:24.959 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:24.691+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:25.376 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:24.973+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:25.376 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:24.973+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:25.376 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:24.973+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:25.376 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:24.973+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:25.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:25.390+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:25.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:25.390+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:25.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:25.390+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:25.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:25.391+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:25.947 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:25.707+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:25.948 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:25.707+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:25.948 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:25.707+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:25.948 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:25.708+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:26.383 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:25.962+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:26.383 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:25.962+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:26.383 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:25.962+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:26.383 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:25.962+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:26.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:26.397+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:26.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:26.397+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:26.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:26.397+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:26.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:26.397+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:26.986 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:26.705+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:26.987 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:26.705+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:26.987 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:26.705+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:26.987 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:26.705+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:27.378 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:27.001+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:27.379 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:27.001+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:27.379 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:27.001+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:27.379 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:27.001+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:27.657 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:27.393+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:27.657 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:27.393+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:27.657 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:27.393+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:27.657 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:27.393+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:27.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:03:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:03:27.671+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 9458 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:03:27.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:27.691+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:27.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:27.691+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:27.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:27.691+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:27.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:27.692+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:28.356 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:28.023+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:28.356 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:28.023+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:28.356 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:28.023+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:28.356 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:28.023+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:28.674 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:28.370+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:28.674 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:28.370+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:28.674 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:28.370+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:28.674 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:28.371+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:28.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:28.688+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:28.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:28.688+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:28.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:28.688+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:28.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:28.689+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:29.317 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:29.066+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:29.317 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:29.066+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:29.317 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:29.066+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:29.317 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:29.067+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:29.669 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:29.330+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:29.670 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:29.330+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:29.670 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:29.330+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:29.670 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:29.330+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:29.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:29.684+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:29.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:29.684+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:29.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:29.684+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:29.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:29.684+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:30.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:30.115+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:30.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:30.115+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:30.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:30.115+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:30.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:30.115+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:30.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:30.289+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:30.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:30.289+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:30.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:30.289+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:30.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:30.289+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:30.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:30.681+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:30.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:30.681+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:30.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:30.681+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:30.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:30.682+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:31.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:31.127+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:31.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:31.127+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:31.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:31.127+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:31.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:31.127+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:31.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:31.309+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:31.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:31.309+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:31.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:31.309+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:31.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:31.309+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:31.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:31.650+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:31.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:31.650+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:31.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:31.650+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:31.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:31.650+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:32.336 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:32.098+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:32.336 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:32.098+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:32.337 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:32.098+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:32.337 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:32.099+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:32.631 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:32.351+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:32.631 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:32.351+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:32.632 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:32.351+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:32.632 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:32.351+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:32.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:03:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:03:32.672+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 9543 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:03:32.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:32.646+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:32.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:32.646+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:32.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:32.646+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:32.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:32.646+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:33.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:33.306+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:33.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:33.306+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:33.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:33.306+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:33.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:33.306+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:33.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:33.073+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:33.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:33.073+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:33.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:33.073+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:33.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:33.073+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:33.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:33.667+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:33.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:33.667+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:33.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:33.667+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:33.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:33.668+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:34.342 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:34.070+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:34.342 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:34.070+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:34.342 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:34.070+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:34.342 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:34.070+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:34.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:34.356+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:34.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:34.356+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:34.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:34.356+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:34.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:34.356+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:35.014 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:34.699+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:35.014 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:34.699+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:35.014 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:34.699+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:35.014 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:34.700+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:35.313 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:35.067+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:35.313 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:35.067+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:35.313 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:35.067+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:35.313 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:35.067+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:35.674 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:35.327+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:35.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:35.327+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:35.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:35.327+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:35.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:35.328+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:35.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:35.689+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:35.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:35.689+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:35.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:35.689+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:35.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:35.689+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:36.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:36.087+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:36.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:36.087+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:36.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:36.087+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:36.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:36.088+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:36.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:36.279+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:36.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:36.279+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:36.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:36.279+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:36.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:36.279+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:37.088 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:36.712+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:37.088 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:36.712+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:37.088 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:36.712+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:37.088 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:36.712+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:37.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:37.102+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:37.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:37.102+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:37.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:37.102+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:37.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:37.103+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:37.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:37.237+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:37.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:37.237+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:37.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:37.237+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:37.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:37.237+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:37.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:03:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:03:37.674+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 9630 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:03:37.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:37.722+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:37.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:37.722+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:37.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:37.722+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:37.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:37.722+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:38.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:38.145+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:38.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:38.145+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:38.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:38.145+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:38.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:38.146+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:38.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:38.233+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:38.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:38.233+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:38.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:38.233+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:38.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:38.233+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:39.155 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:38.744+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:39.155 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:38.744+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:39.156 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:38.744+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:39.156 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:38.745+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:39.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:39.170+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:39.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:39.170+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:39.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:39.170+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:39.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:39.170+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:39.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:39.263+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:39.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:39.263+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:39.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:39.263+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:39.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:39.263+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:40.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:39.736+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:40.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:39.736+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:40.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:39.736+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:40.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:39.736+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:40.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:40.220+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:40.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:40.220+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:40.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:40.220+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:40.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:40.220+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:40.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:40.273+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:40.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:40.273+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:40.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:40.273+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:40.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:40.273+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:41.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:40.744+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:41.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:40.744+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:41.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:40.745+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:41.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:40.745+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:41.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:41.214+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:41.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:41.214+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:41.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:41.214+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:41.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:41.214+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:41.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:41.234+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:41.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:41.234+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:41.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:41.234+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:41.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:41.234+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:42.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:41.792+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:42.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:41.792+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:42.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:41.792+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:42.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:41.793+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:42.662 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:42.262+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:42.662 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:42.262+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:42.663 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:42.262+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:42.663 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:42.262+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:42.663 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:42.277+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:42.663 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:42.277+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:42.663 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:42.277+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:42.663 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:42.277+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:42.926 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:03:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:03:42.676+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 9714 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:03:42.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:42.761+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:42.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:42.761+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:42.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:42.761+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:42.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:42.762+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:43.555 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:43.232+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:43.556 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:43.232+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:43.556 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:43.232+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:43.556 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:43.232+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:43.556 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:43.269+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:43.556 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:43.269+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:43.556 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:43.269+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:43.556 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:43.269+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:43.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:43.717+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:43.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:43.717+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:43.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:43.717+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:43.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:43.718+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:44.673 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:44.261+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:44.673 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:44.261+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:44.673 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:44.261+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:44.674 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:44.262+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:44.674 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:44.236+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:44.674 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:44.236+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:44.674 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:44.236+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:44.674 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:44.236+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:44.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:44.688+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:44.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:44.688+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:44.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:44.688+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:44.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:44.688+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:45.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:45.189+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:45.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:45.189+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:45.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:45.189+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:45.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:45.189+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:45.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:45.220+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:45.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:45.220+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:45.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:45.220+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:45.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:45.221+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:46.142 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:45.704+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:46.142 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:45.704+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:46.142 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:45.704+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:46.142 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:45.704+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:46.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:46.156+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:46.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:46.156+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:46.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:46.156+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:46.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:46.156+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:46.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:46.212+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:46.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:46.212+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:46.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:46.212+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:46.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:46.212+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:46.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:46.678+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:46.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:46.678+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:46.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:46.678+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:46.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:46.678+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:47.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:47.229+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:47.427 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:47.229+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:47.427 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:47.229+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:47.427 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:47.229+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:47.427 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:47.160+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:47.427 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:47.160+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:47.427 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:47.160+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:47.427 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:47.160+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:47.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:03:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:03:47.678+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 9800 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:03:47.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:47.676+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:47.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:47.676+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:47.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:47.676+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:47.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:47.677+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:48.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:48.261+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:48.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:48.261+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:48.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:48.261+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:48.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:48.261+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:48.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:48.180+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:48.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:48.180+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:48.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:48.180+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:48.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:48.181+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:48.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:48.661+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:48.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:48.661+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:48.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:48.662+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:48.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:48.662+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:49.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:49.154+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:49.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:49.154+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:49.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:49.154+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:49.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:49.155+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:49.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:49.244+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:49.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:49.244+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:49.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:49.244+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:49.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:49.244+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:49.964 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:49.699+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:49.964 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:49.699+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:49.964 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:49.699+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:49.964 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:49.699+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:50.214 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:50.181+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:50.215 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:50.181+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:50.215 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:50.181+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:50.215 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:50.181+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:50.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:50.229+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:50.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:50.229+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:50.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:50.229+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:50.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:50.229+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:51.150 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:50.693+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:51.150 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:50.693+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:51.150 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:50.693+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:51.150 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:50.694+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:51.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:51.164+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:51.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:51.164+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:51.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:51.164+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:51.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:51.165+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:51.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:51.257+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:51.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:51.257+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:51.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:51.257+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:51.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:51.257+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:52.000 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:51.702+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:52.000 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:51.702+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:52.000 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:51.702+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:52.000 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:51.702+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:52.272 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:52.168+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:52.272 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:52.168+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:52.272 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:52.168+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:52.272 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:52.168+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:52.653 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:52.286+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:52.653 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:52.286+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:52.653 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:52.286+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:52.653 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:52.286+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:52.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:03:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:03:52.680+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 9887 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:03:52.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:52.667+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:52.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:52.667+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:52.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:52.667+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:52.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:52.668+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:53.557 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:53.203+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:53.557 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:53.203+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:53.557 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:53.203+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:53.557 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:53.203+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:53.557 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:53.309+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:53.557 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:53.309+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:53.557 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:53.309+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:53.557 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:53.309+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:53.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:53.668+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:53.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:53.668+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:53.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:53.668+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:53.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:53.669+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:54.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:54.191+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:54.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:54.191+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:54.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:54.191+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:54.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:54.191+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:54.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:54.305+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:54.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:54.305+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:54.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:54.305+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:54.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:54.305+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:55.014 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:54.691+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:55.014 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:54.691+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:55.014 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:54.691+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:55.014 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:54.692+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:55.313 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:55.174+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:55.313 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:55.174+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:55.313 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:55.174+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:55.313 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:55.174+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:55.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:55.328+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:55.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:55.328+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:55.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:55.328+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:55.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:55.328+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:56.172 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:55.739+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:56.172 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:55.739+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:56.173 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:55.739+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:56.173 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:55.739+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:56.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:56.187+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:56.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:56.187+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:56.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:56.187+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:56.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:56.187+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:56.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:56.340+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:56.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:56.340+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:56.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:56.340+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:56.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:56.340+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:56.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:56.697+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:56.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:56.697+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:56.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:56.697+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:56.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:56.697+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:57.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:57.168+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:57.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:57.168+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:57.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:57.168+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:57.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:57.168+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:57.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:57.309+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:57.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:57.309+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:57.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:57.309+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:57.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:57.309+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:57.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:03:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:03:57.682+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 9982 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:03:57.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:57.661+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:57.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:57.661+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:57.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:57.661+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:57.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:57.662+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:58.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:58.183+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:58.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:58.183+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:58.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:58.183+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:58.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:58.183+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:58.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:58.294+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:58.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:58.294+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:58.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:58.294+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:58.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:58.295+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:58.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:58.666+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:58.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:58.666+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:58.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:58.666+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:58.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:58.666+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:03:59.625 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:59.219+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:59.625 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:59.219+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:59.625 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:59.219+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:03:59.625 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:03:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:03:59.220+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:59.625 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:59.296+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:59.626 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:59.296+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:59.626 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:59.296+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:03:59.626 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:03:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:03:59.296+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:03:59.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:59.640+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:59.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:59.640+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:59.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:59.640+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:03:59.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:03:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:03:59.640+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:00.326 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:00.178+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:00.326 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:00.178+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:00.326 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:00.178+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:00.326 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:00.178+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:00.579 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:00.340+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:00.580 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:00.340+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:00.580 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:00.340+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:00.580 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:00.340+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:00.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:00.594+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:00.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:00.594+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:00.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:00.594+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:00.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:00.594+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:01.587 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:01.314+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:01.587 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:01.314+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:01.587 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:01.314+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:01.587 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:01.314+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:01.588 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:01.211+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:01.588 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:01.211+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:01.588 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:01.211+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:01.588 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:01.211+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:01.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:01.602+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:01.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:01.602+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:01.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:01.602+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:01.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:01.602+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:02.267 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:02.225+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:02.267 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:02.225+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:02.267 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:02.225+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:02.267 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:02.225+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:02.544 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:02.282+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:02.544 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:02.282+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:02.544 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:02.282+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:02.544 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:02.282+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:02.796 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:04:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:04:02.684+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 10077 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:04:02.796 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:02.559+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:02.796 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:02.559+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:02.796 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:02.559+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:02.796 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:02.559+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:03.523 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:03.224+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:03.523 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:03.224+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:03.523 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:03.224+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:03.523 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:03.224+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:03.523 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:03.267+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:03.523 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:03.267+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:03.523 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:03.267+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:03.524 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:03.267+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:03.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:03.538+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:03.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:03.538+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:03.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:03.538+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:03.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:03.538+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:04.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:04.188+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:04.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:04.188+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:04.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:04.188+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:04.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:04.188+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:04.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:04.287+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:04.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:04.287+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:04.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:04.287+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:04.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:04.287+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:04.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:04.567+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:04.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:04.567+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:04.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:04.567+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:04.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:04.567+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:05.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:05.275+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:05.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:05.275+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:05.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:05.275+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:05.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:05.275+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:05.427 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:05.237+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:05.427 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:05.237+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:05.427 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:05.237+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:05.427 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:05.237+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:05.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:05.566+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:05.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:05.566+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:05.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:05.566+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:05.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:05.566+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:06.518 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:06.216+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:06.518 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:06.216+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:06.519 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:06.216+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:06.519 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:06.217+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:06.519 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:06.229+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:06.519 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:06.229+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:06.519 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:06.229+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:06.519 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:06.229+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:06.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:06.531+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:06.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:06.531+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:06.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:06.531+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:06.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:06.532+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:07.534 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:07.211+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:07.534 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:07.211+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:07.534 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:07.211+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:07.534 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:07.211+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:07.534 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:07.223+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:07.535 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:07.223+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:07.535 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:07.223+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:07.535 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:07.223+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:07.798 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:04:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:04:07.685+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 10174 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:04:07.798 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:07.548+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:07.798 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:07.548+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:07.798 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:07.548+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:07.798 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:07.548+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:08.508 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:08.194+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:08.508 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:08.194+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:08.508 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:08.194+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:08.508 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:08.194+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:08.509 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:08.221+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:08.509 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:08.221+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:08.509 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:08.221+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:08.509 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:08.221+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:08.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:08.522+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:08.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:08.522+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:08.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:08.522+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:08.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:08.523+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:09.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:09.166+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:09.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:09.166+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:09.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:09.166+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:09.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:09.167+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:09.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:09.199+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:09.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:09.199+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:09.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:09.199+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:09.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:09.200+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:09.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:09.549+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:09.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:09.549+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:09.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:09.549+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:09.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:09.549+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:10.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:10.177+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:10.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:10.177+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:10.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:10.177+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:10.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:10.178+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:10.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:10.223+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:10.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:10.223+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:10.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:10.223+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:10.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:10.223+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:10.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:10.571+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:10.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:10.571+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:10.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:10.571+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:10.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:10.572+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:11.539 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:11.273+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:11.539 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:11.273+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:11.540 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:11.273+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:11.540 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:11.273+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:11.540 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:11.212+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:11.540 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:11.212+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:11.540 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:11.212+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:11.540 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:11.212+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:11.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:11.553+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:11.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:11.553+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:11.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:11.553+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:11.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:11.554+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:12.270 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:12.250+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:12.270 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:12.250+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:12.270 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:12.250+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:12.270 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:12.250+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:12.673 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:12.525+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:12.674 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:12.525+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:12.674 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:12.525+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:12.674 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:12.525+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:12.674 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:12.284+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:12.674 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:12.284+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:12.674 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:12.284+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:12.674 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:12.285+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:12.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:04:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:04:12.688+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 10266 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:04:13.558 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:13.204+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:13.558 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:13.204+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:13.558 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:13.204+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:13.558 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:13.204+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:13.559 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:13.322+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:13.559 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:13.322+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:13.559 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:13.322+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:13.559 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:13.322+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:13.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:13.575+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:13.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:13.575+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:13.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:13.575+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:13.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:13.576+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:14.523 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:14.195+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:14.523 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:14.195+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:14.523 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:14.195+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:14.523 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:14.195+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:14.523 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:14.314+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:14.524 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:14.314+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:14.524 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:14.314+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:14.524 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:14.315+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:14.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:14.537+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:14.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:14.537+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:14.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:14.537+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:14.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:14.538+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:15.215 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:15.167+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:15.215 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:15.167+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:15.215 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:15.167+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:15.215 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:15.167+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:15.549 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:15.355+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:15.549 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:15.355+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:15.549 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:15.355+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:15.549 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:15.355+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:15.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:15.563+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:15.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:15.563+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:15.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:15.563+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:15.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:15.564+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:16.596 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:16.366+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:16.597 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:16.366+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:16.597 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:16.366+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:16.597 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:16.367+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:16.597 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:16.213+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:16.597 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:16.213+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:16.597 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:16.213+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:16.597 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:16.214+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:16.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:16.611+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:16.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:16.611+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:16.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:16.611+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:16.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:16.611+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:17.644 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:17.325+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:17.644 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:17.325+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:17.644 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:17.325+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:17.644 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:17.325+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:17.644 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:17.249+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:17.644 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:17.249+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:17.644 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:17.249+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:17.645 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:17.249+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:17.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:04:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:04:17.690+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 10363 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:04:17.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:17.658+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:17.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:17.658+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:17.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:17.658+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:17.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:17.659+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:18.323 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:18.204+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:18.323 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:18.204+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:18.323 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:18.204+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:18.323 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:18.205+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:18.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:18.337+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:18.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:18.337+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:18.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:18.337+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:18.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:18.338+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:19.144 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:18.699+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:19.144 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:18.699+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:19.144 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:18.699+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:19.144 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:18.700+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:19.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:19.290+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:19.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:19.290+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:19.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:19.290+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:19.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:19.290+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:19.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:19.158+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:19.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:19.158+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:19.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:19.158+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:19.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:19.158+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:19.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:19.657+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:19.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:19.657+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:19.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:19.657+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:19.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:19.657+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:20.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:20.269+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:20.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:20.269+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:20.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:20.269+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:20.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:20.270+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:20.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:20.140+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:20.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:20.140+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:20.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:20.140+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:20.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:20.140+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:20.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:20.607+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:20.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:20.607+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:20.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:20.607+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:20.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:20.608+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:21.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:21.236+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:21.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:21.236+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:21.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:21.236+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:21.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:21.237+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:21.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:21.120+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:21.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:21.120+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:21.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:21.120+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:21.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:21.120+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:21.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:21.617+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:21.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:21.617+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:21.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:21.617+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:21.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:21.617+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:22.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:22.206+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:22.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:22.206+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:22.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:22.206+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:22.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:22.206+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:22.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:22.160+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:22.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:22.160+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:22.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:22.160+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:22.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:22.160+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:22.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:22.591+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:22.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:22.591+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:22.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:22.591+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:22.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:22.592+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:23.150 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:04:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:04:22.691+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 10457 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:04:23.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:23.165+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:23.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:23.165+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:23.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:23.165+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:23.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:23.165+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:23.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:23.224+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:23.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:23.224+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:23.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:23.224+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:23.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:23.224+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:23.615+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:23.615+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:23.615+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:23.615+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:24.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:24.142+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:24.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:24.142+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:24.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:24.142+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:24.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:24.142+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:24.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:24.254+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:24.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:24.254+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:24.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:24.254+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:24.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:24.255+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:24.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:24.656+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:24.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:24.656+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:24.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:24.656+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:24.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:24.657+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:25.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:25.118+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:25.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:25.118+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:25.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:25.118+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:25.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:25.119+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:25.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:25.282+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:25.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:25.282+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:25.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:25.282+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:25.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:25.282+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:25.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:25.646+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:25.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:25.646+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:25.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:25.646+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:25.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:25.646+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:26.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:26.110+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:26.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:26.110+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:26.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:26.110+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:26.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:26.111+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:26.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:26.265+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:26.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:26.265+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:26.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:26.265+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:26.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:26.265+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:26.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:26.625+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:26.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:26.625+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:26.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:26.625+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:26.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:26.626+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:27.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:27.087+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:27.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:27.087+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:27.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:27.087+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:27.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:27.087+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:27.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:27.305+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:27.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:27.305+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:27.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:27.305+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:27.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:27.305+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:27.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:04:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:04:27.693+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 10553 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:04:27.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:27.669+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:27.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:27.669+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:27.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:27.669+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:27.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:27.669+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:28.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:28.135+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:28.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:28.135+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:28.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:28.135+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:28.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:28.135+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:28.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:28.265+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:28.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:28.265+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:28.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:28.265+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:28.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:28.265+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:28.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:28.669+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:28.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:28.669+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:28.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:28.669+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:28.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:28.669+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:29.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:29.153+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:29.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:29.153+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:29.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:29.153+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:29.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:29.154+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:29.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:29.231+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:29.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:29.231+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:29.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:29.231+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:29.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:29.232+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:30.141 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:29.709+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:30.142 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:29.710+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:30.142 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:29.710+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:30.142 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:29.710+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:30.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:30.244+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:30.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:30.244+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:30.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:30.244+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:30.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:30.244+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:30.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:30.156+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:30.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:30.156+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:30.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:30.156+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:30.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:30.156+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:31.107 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:30.743+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:31.107 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:30.743+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:31.107 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:30.743+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:31.107 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:30.743+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:31.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:31.228+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:31.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:31.228+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:31.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:31.228+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:31.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:31.228+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:31.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:31.121+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:31.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:31.121+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:31.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:31.121+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:31.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:31.121+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:32.000 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:31.702+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:32.000 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:31.702+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:32.000 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:31.702+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:32.000 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:31.702+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:32.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:32.235+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:32.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:32.235+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:32.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:32.235+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:32.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:32.235+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:32.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:32.098+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:32.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:32.098+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:32.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:32.098+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:32.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:32.098+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:33.079 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:04:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:04:32.695+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 10645 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:04:33.079 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:32.728+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:33.080 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:32.728+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:33.080 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:32.728+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:33.080 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:32.728+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:33.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:33.094+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:33.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:33.094+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:33.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:33.094+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:33.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:33.094+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:33.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:33.233+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:33.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:33.233+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:33.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:33.233+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:33.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:33.233+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:33.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:33.683+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:33.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:33.683+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:33.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:33.683+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:33.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:33.683+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:34.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:34.081+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:34.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:34.081+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:34.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:34.081+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:34.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:34.082+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:34.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:34.192+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:34.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:34.192+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:34.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:34.192+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:34.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:34.192+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:34.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:34.669+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:34.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:34.669+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:34.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:34.669+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:34.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:34.669+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:35.414 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:35.180+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:35.414 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:35.180+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:35.414 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:35.180+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:35.414 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:35.180+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:35.414 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:35.122+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:35.414 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:35.122+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:35.414 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:35.122+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:35.414 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:35.123+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:35.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:35.666+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:35.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:35.666+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:35.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:35.666+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:35.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:35.667+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:36.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:36.184+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:36.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:36.184+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:36.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:36.184+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:36.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:36.184+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:36.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:36.083+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:36.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:36.083+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:36.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:36.083+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:36.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:36.083+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:36.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:36.674+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:36.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:36.674+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:36.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:36.674+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:36.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:36.674+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:37.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:37.196+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:37.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:37.196+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:37.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:37.196+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:37.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:37.196+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:37.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:37.103+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:37.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:37.103+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:37.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:37.103+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:37.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:37.103+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:37.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:04:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:04:37.698+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 10740 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:04:37.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:37.661+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:37.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:37.661+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:37.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:37.661+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:37.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:37.662+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:38.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:38.212+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:38.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:38.212+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:38.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:38.212+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:38.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:38.212+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:38.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:38.107+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:38.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:38.107+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:38.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:38.107+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:38.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:38.108+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:38.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:38.663+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:38.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:38.663+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:38.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:38.663+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:38.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:38.663+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:39.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:39.238+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:39.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:39.238+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:39.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:39.238+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:39.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:39.238+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:39.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:39.112+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:39.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:39.112+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:39.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:39.112+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:39.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:39.112+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:39.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:39.649+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:39.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:39.649+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:39.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:39.649+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:39.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:39.649+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:40.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:40.229+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:40.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:40.229+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:40.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:40.229+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:40.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:40.229+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:40.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:40.141+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:40.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:40.141+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:40.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:40.141+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:40.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:40.141+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:40.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:40.690+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:40.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:40.690+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:40.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:40.690+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:40.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:40.691+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:41.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:41.187+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:41.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:41.187+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:41.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:41.187+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:41.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:41.187+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:41.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:41.121+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:41.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:41.121+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:41.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:41.121+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:41.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:41.121+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:42.000 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:41.693+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:42.000 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:41.693+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:42.000 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:41.693+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:42.001 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:41.693+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:42.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:42.193+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:42.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:42.193+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:42.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:42.193+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:42.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:42.193+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:42.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:42.122+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:42.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:42.122+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:42.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:42.122+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:42.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:42.122+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:42.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:04:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:04:42.699+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 10833 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:04:42.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:42.699+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:42.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:42.700+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:42.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:42.700+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:42.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:42.700+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:43.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:43.216+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:43.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:43.216+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:43.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:43.216+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:43.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:43.216+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:43.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:43.128+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:43.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:43.128+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:43.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:43.128+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:43.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:43.128+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:43.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:43.675+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:43.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:43.675+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:43.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:43.675+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:43.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:43.675+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:44.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:44.222+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:44.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:44.222+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:44.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:44.222+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:44.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:44.222+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:44.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:44.115+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:44.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:44.115+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:44.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:44.115+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:44.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:44.116+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:44.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:44.652+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:44.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:44.652+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:44.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:44.652+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:44.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:44.653+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:45.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:45.206+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:45.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:45.206+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:45.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:45.206+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:45.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:45.206+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:45.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:45.143+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:45.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:45.143+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:45.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:45.143+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:45.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:45.143+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:45.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:45.653+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:45.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:45.653+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:45.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:45.653+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:45.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:45.653+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:46.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:46.244+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:46.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:46.244+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:46.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:46.244+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:46.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:46.244+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:46.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:46.180+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:46.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:46.180+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:46.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:46.180+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:46.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:46.181+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:46.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:46.685+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:46.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:46.685+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:46.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:46.685+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:46.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:46.686+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:47.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:47.283+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:47.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:47.283+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:47.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:47.283+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:47.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:47.283+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:47.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:47.171+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:47.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:47.171+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:47.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:47.171+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:47.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:47.171+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:48.136 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:04:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:04:47.701+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 10928 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:04:48.136 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:47.732+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:48.136 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:47.732+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:48.136 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:47.732+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:48.136 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:47.733+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:48.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:48.306+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:48.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:48.306+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:48.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:48.306+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:48.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:48.306+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:48.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:48.151+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:48.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:48.151+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:48.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:48.151+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:48.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:48.151+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:49.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:48.694+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:49.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:48.694+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:49.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:48.694+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:49.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:48.694+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:49.650 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:49.189+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:49.650 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:49.189+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:49.650 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:49.189+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:49.650 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:49.190+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:49.651 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:49.288+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:49.651 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:49.288+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:49.651 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:49.288+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:49.651 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:49.288+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:49.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:49.665+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:49.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:49.665+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:49.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:49.665+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:49.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:49.665+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:50.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:50.338+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:50.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:50.338+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:50.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:50.338+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:50.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:50.338+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:50.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:50.223+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:50.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:50.223+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:50.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:50.223+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:50.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:50.223+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:50.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:50.657+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:50.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:50.657+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:50.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:50.657+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:50.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:50.657+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:51.615 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:51.200+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:51.615 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:51.200+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:51.615 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:51.200+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:51.615 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:51.200+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:51.615 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:51.332+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:51.615 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:51.332+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:51.615 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:51.332+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:51.615 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:51.332+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:51.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:51.630+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:51.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:51.630+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:51.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:51.630+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:51.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:51.630+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:52.318 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:52.235+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:52.318 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:52.235+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:52.318 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:52.235+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:52.318 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:52.235+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:52.586 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:52.332+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:52.586 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:52.332+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:52.587 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:52.332+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:52.587 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:52.333+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:52.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:04:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:04:52.704+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 11020 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:04:52.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:52.601+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:52.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:52.601+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:52.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:52.601+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:52.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:52.601+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:53.561 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:53.344+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:53.561 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:53.344+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:53.562 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:53.344+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:53.562 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:53.344+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:53.562 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:53.259+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:53.562 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:53.259+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:53.562 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:53.259+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:53.562 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:53.259+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:53.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:53.624+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:53.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:53.624+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:53.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:53.624+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:53.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:53.624+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:54.642 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:54.351+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:54.642 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:54.351+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:54.642 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:54.351+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:54.642 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:54.352+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:54.643 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:54.221+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:54.643 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:54.221+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:54.643 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:54.221+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:54.643 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:54.221+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:54.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:54.656+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:54.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:54.656+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:54.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:54.656+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:54.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:54.657+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:55.613 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:55.268+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:55.614 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:55.268+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:55.614 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:55.268+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:55.614 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:55.268+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:55.614 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:55.360+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:55.614 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:55.360+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:55.614 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:55.360+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:55.614 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:55.360+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:55.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:55.637+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:55.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:55.638+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:55.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:55.638+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:55.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:55.638+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:56.591 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:56.384+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:56.591 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:56.384+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:56.591 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:56.384+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:56.591 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:56.384+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:56.591 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:56.252+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:56.592 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:56.252+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:56.592 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:56.252+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:56.592 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:56.252+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:56.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:56.604+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:56.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:56.604+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:56.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:56.604+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:56.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:56.605+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:57.571 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:57.343+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:57.571 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:57.343+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:57.571 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:57.343+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:57.571 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:57.343+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:57.571 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:57.228+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:57.571 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:57.228+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:57.571 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:57.228+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:57.572 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:57.228+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:57.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:04:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:04:57.706+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 11117 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:04:57.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:57.585+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:57.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:57.585+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:57.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:57.585+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:57.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:57.585+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:58.559 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:58.341+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:58.559 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:58.341+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:58.559 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:58.341+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:58.559 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:58.341+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:58.560 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:58.216+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:58.560 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:58.216+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:58.560 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:58.216+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:58.560 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:58.216+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:58.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:58.573+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:58.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:58.573+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:58.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:58.573+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:58.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:58.574+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:04:59.562 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:59.357+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:59.562 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:59.358+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:59.563 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:59.358+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:04:59.563 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:04:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:04:59.358+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:59.563 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:59.251+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:59.563 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:59.251+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:59.563 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:59.251+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:04:59.563 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:04:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:04:59.251+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:04:59.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:59.576+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:59.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:59.576+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:59.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:59.576+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:04:59.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:04:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:04:59.577+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:00.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:00.347+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:00.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:00.347+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:00.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:00.347+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:00.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:00.347+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:00.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:00.278+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:00.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:00.278+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:00.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:00.278+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:00.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:00.279+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:00.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:00.621+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:00.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:00.621+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:00.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:00.621+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:00.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:00.621+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:01.373 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:01.275+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:01.374 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:01.275+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:01.374 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:01.275+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:01.374 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:01.275+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:01.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:01.388+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:01.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:01.388+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:01.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:01.388+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:01.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:01.388+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:01.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:01.614+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:01.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:01.614+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:01.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:01.614+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:01.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:01.614+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:02.565 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:02.431+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:02.565 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:02.431+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:02.565 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:02.431+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:02.565 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:02.432+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:02.565 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:02.324+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:02.565 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:02.324+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:02.565 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:02.324+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:02.565 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:02.324+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:02.816 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:05:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:05:02.708+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 11210 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:05:02.817 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:02.579+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:02.817 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:02.579+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:02.817 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:02.579+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:02.817 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:02.579+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:03.562 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:03.473+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:03.562 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:03.473+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:03.562 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:03.473+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:03.562 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:03.473+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:03.563 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:03.297+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:03.563 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:03.297+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:03.563 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:03.297+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:03.563 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:03.298+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:03.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:03.614+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:03.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:03.614+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:03.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:03.614+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:03.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:03.614+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:04.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:04.301+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:04.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:04.301+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:04.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:04.301+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:04.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:04.301+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:04.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:04.484+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:04.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:04.484+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:04.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:04.484+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:04.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:04.485+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:04.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:04.658+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:04.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:04.658+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:04.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:04.658+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:04.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:04.658+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:05.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:05.494+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:05.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:05.494+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:05.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:05.494+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:05.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:05.495+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:05.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:05.252+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:05.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:05.252+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:05.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:05.252+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:05.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:05.252+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:06.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:05.697+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:06.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:05.697+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:06.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:05.697+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:06.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:05.698+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:06.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:06.462+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:06.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:06.462+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:06.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:06.462+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:06.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:06.463+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:06.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:06.265+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:06.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:06.265+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:06.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:06.265+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:06.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:06.265+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:07.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:06.696+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:07.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:06.696+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:07.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:06.696+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:07.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:06.696+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:07.660 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:07.491+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:07.661 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:07.491+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:07.661 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:07.491+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:07.662 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:07.491+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:07.662 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:07.282+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:07.667 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:07.282+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:07.668 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:07.282+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:07.668 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:07.283+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:07.927 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:05:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:05:07.710+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 11305 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:05:07.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:07.673+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:07.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:07.673+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:07.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:07.673+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:07.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:07.674+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:08.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:08.518+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:08.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:08.518+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:08.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:08.518+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:08.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:08.518+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:08.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:08.304+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:08.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:08.304+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:08.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:08.304+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:08.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:08.304+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:09.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:08.705+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:09.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:08.705+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:09.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:08.705+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:09.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:08.705+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:09.523 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:09.270+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:09.523 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:09.270+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:09.523 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:09.270+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:09.524 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:09.271+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:09.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:09.537+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:09.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:09.537+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:09.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:09.537+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:09.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:09.538+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:09.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:09.695+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:09.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:09.695+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:09.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:09.695+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:09.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:09.695+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:10.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:10.231+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:10.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:10.231+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:10.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:10.231+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:10.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:10.231+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:10.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:10.561+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:10.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:10.561+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:10.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:10.561+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:10.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:10.561+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:10.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:10.684+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:10.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:10.684+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:10.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:10.684+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:10.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:10.685+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:11.591 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:11.263+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:11.591 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:11.263+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:11.591 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:11.263+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:11.591 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:11.264+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:11.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:11.605+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:11.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:11.605+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:11.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:11.605+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:11.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:11.606+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:11.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:11.730+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:11.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:11.730+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:11.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:11.730+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:11.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:11.730+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:12.561 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:12.291+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:12.561 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:12.291+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:12.561 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:12.291+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:12.561 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:12.291+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:12.822 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:05:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:05:12.711+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 11398 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:05:12.822 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:12.575+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:12.822 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:12.575+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:12.822 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:12.575+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:12.823 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:12.575+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:12.823 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:12.698+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:12.823 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:12.698+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:12.823 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:12.698+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:12.823 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:12.698+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:13.578 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:13.337+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:13.578 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:13.337+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:13.578 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:13.337+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:13.578 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:13.337+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:13.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:13.592+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:13.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:13.592+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:13.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:13.592+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:13.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:13.593+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:13.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:13.678+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:13.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:13.678+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:13.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:13.678+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:13.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:13.679+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:14.571 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:14.308+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:14.571 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:14.308+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:14.571 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:14.308+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:14.571 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:14.308+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:14.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:14.585+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:14.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:14.585+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:14.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:14.585+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:14.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:14.585+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:14.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:14.660+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:14.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:14.660+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:14.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:14.660+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:14.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:14.661+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:15.577 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:15.306+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:15.577 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:15.306+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:15.577 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:15.306+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:15.577 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:15.306+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:15.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:15.592+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:15.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:15.592+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:15.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:15.592+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:15.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:15.592+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:15.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:15.658+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:15.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:15.658+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:15.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:15.658+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:15.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:15.659+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:16.537 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:16.263+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:16.537 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:16.263+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:16.538 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:16.263+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:16.538 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:16.263+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:16.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:16.552+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:16.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:16.552+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:16.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:16.552+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:16.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:16.552+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:16.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:16.672+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:16.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:16.672+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:16.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:16.672+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:16.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:16.673+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:17.553 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:17.230+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:17.553 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:17.230+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:17.553 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:17.230+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:17.553 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:17.230+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:17.825 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:05:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:05:17.714+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 11494 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:05:17.825 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:17.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:17.825 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:17.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:17.825 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:17.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:17.825 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:17.568+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:17.825 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:17.682+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:17.826 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:17.682+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:17.826 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:17.682+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:17.826 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:17.682+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:18.593 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:18.201+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:18.593 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:18.201+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:18.593 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:18.201+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:18.593 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:18.201+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:18.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:18.607+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:18.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:18.607+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:18.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:18.607+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:18.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:18.607+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:18.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:18.696+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:18.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:18.696+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:18.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:18.696+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:18.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:18.696+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:19.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:19.188+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:19.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:19.188+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:19.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:19.188+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:19.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:19.188+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:19.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:19.651+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:19.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:19.651+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:19.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:19.651+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:19.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:19.652+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:19.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:19.673+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:19.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:19.673+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:19.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:19.673+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:19.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:19.673+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:20.617 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:20.220+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:20.617 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:20.220+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:20.618 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:20.220+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:20.618 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:20.220+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:20.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:20.653+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:20.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:20.653+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:20.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:20.653+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:20.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:20.653+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:20.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:20.632+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:20.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:20.632+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:20.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:20.632+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:20.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:20.632+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:21.613 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:21.265+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:21.613 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:21.265+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:21.613 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:21.265+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:21.613 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:21.265+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:21.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:21.700+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:21.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:21.700+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:21.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:21.700+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:21.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:21.700+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:21.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:21.628+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:21.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:21.628+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:21.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:21.628+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:21.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:21.628+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:22.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:22.258+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:22.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:22.258+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:22.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:22.258+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:22.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:22.259+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:22.682 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:22.639+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:22.682 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:22.639+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:22.682 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:22.639+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:22.683 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:22.639+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:23.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:05:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:05:22.716+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 11587 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:05:23.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:22.696+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:23.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:22.696+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:23.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:22.696+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:23.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:22.697+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:23.563 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:23.246+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:23.564 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:23.246+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:23.564 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:23.246+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:23.564 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:23.246+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:23.598+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:23.598+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:23.598+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:23.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:23.598+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:23.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:23.734+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:23.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:23.735+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:23.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:23.735+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:23.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:23.735+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:24.612 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:24.226+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:24.612 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:24.226+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:24.612 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:24.226+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:24.612 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:24.226+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:24.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:24.768+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:24.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:24.768+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:24.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:24.768+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:24.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:24.768+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:24.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:24.627+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:24.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:24.627+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:24.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:24.627+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:24.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:24.627+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:25.606 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:25.273+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:25.606 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:25.273+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:25.606 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:25.273+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:25.606 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:25.274+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:25.913 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:25.724+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:25.913 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:25.724+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:25.913 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:25.724+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:25.914 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:25.725+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:25.914 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:25.621+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:25.914 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:25.621+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:25.914 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:25.621+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:25.914 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:25.621+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:26.625 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:26.300+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:26.625 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:26.300+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:26.625 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:26.300+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:26.625 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:26.300+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:26.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:26.640+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:26.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:26.640+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:26.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:26.640+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:26.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:26.640+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:26.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:26.717+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:26.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:26.717+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:26.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:26.717+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:26.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:26.717+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:27.660 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:27.329+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:27.660 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:27.329+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:27.660 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:27.329+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:27.660 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:27.329+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:27.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:05:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:05:27.718+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 11682 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:05:27.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:27.674+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:27.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:27.674+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:27.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:27.674+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:27.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:27.675+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:27.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:27.765+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:27.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:27.766+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:27.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:27.766+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:27.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:27.766+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:28.619 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:28.325+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:28.620 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:28.325+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:28.620 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:28.325+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:28.620 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:28.325+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:28.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:28.788+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:28.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:28.788+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:28.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:28.788+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:28.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:28.788+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:28.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:28.634+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:28.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:28.634+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:28.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:28.634+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:28.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:28.634+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:29.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:29.594+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:29.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:29.594+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:29.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:29.594+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:29.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:29.594+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:29.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:29.357+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:29.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:29.357+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:29.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:29.357+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:29.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:29.357+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:30.130 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:29.752+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:30.131 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:29.752+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:30.131 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:29.752+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:30.131 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:29.752+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:30.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:30.357+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:30.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:30.357+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:30.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:30.357+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:30.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:30.357+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:30.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:30.794+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:30.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:30.794+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:30.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:30.794+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:30.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:30.794+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:30.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:30.638+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:30.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:30.638+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:30.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:30.638+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:30.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:30.638+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:31.628 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:31.329+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:31.628 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:31.329+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:31.628 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:31.329+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:31.629 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:31.329+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:31.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:31.769+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:31.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:31.769+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:31.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:31.769+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:31.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:31.769+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:31.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:31.643+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:31.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:31.643+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:31.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:31.643+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:31.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:31.643+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:32.595 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:32.280+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:32.595 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:32.280+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:32.595 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:32.280+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:32.595 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:32.280+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:32.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:05:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:05:32.720+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 11776 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:05:32.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:32.610+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:32.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:32.610+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:32.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:32.610+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:32.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:32.610+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:32.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:32.817+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:32.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:32.817+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:32.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:32.817+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:32.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:32.817+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:33.564 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:33.322+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:33.564 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:33.322+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:33.564 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:33.322+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:33.564 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:33.322+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:33.837 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:33.602+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:33.837 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:33.602+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:33.837 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:33.602+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:33.837 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:33.602+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:34.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:33.851+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:34.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:33.851+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:34.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:33.851+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:34.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:33.851+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:34.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:34.367+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:34.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:34.367+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:34.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:34.367+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:34.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:34.367+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:34.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:34.592+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:34.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:34.592+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:34.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:34.592+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:34.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:34.593+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:35.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:34.833+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:35.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:34.833+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:35.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:34.833+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:35.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:34.833+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:35.588 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:35.353+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:35.589 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:35.353+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:35.589 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:35.353+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:35.589 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:35.353+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:35.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:35.840+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:35.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:35.840+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:35.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:35.840+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:35.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:35.840+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:35.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:35.603+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:35.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:35.603+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:35.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:35.603+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:35.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:35.603+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:36.614 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:36.377+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:36.615 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:36.377+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:36.615 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:36.377+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:36.615 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:36.377+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:36.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:36.798+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:36.958 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:36.798+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:36.958 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:36.798+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:36.958 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:36.799+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:36.959 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:36.629+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:36.959 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:36.629+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:36.959 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:36.629+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:36.959 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:36.629+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:37.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:37.627+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:37.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:37.627+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:37.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:37.627+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:37.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:37.627+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:37.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:37.398+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:37.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:37.398+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:37.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:37.398+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:37.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:37.398+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:38.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:05:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:05:37.722+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 11873 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:05:38.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:37.799+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:38.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:37.799+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:38.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:37.799+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:38.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:37.799+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:38.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:38.609+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:38.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:38.609+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:38.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:38.609+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:38.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:38.610+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:38.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:38.416+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:38.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:38.416+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:38.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:38.416+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:38.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:38.416+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:39.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:38.819+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:39.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:38.819+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:39.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:38.819+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:39.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:38.820+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:39.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:39.633+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:39.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:39.633+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:39.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:39.633+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:39.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:39.634+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:39.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:39.418+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:39.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:39.418+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:39.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:39.418+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:39.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:39.418+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:40.131 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:39.860+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:40.131 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:39.860+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:40.131 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:39.860+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:40.131 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:39.860+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:40.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:40.646+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:40.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:40.646+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:40.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:40.646+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:40.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:40.646+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:40.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:40.423+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:40.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:40.423+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:40.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:40.423+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:40.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:40.423+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:41.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:40.838+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:41.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:40.838+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:41.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:40.838+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:41.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:40.838+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:41.644 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:41.383+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:41.644 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:41.383+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:41.644 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:41.383+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:41.644 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:41.383+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:41.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:41.819+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:41.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:41.819+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:41.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:41.819+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:41.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:41.820+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:41.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:41.659+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:41.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:41.659+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:41.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:41.659+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:41.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:41.659+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:42.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:42.624+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:42.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:42.624+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:42.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:42.624+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:42.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:42.624+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:42.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:42.381+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:42.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:42.381+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:42.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:42.381+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:42.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:42.381+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:43.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:05:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:05:42.724+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 11968 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:05:43.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:42.867+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:43.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:42.867+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:43.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:42.867+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:43.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:42.867+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:43.643 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:43.352+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:43.644 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:43.352+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:43.644 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:43.352+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:43.644 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:43.352+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:43.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:43.860+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:43.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:43.860+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:43.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:43.860+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:43.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:43.860+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:43.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:43.658+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:43.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:43.658+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:43.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:43.658+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:43.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:43.658+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:44.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:44.358+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:44.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:44.358+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:44.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:44.358+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:44.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:44.358+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:45.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:44.894+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:45.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:44.894+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:45.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:44.894+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:45.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:44.894+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:45.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:44.698+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:45.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:44.698+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:45.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:44.698+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:45.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:44.698+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:45.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:45.385+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:45.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:45.385+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:45.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:45.385+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:45.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:45.385+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:46.112 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:45.921+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:46.113 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:45.921+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:46.113 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:45.921+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:46.113 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:45.921+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:46.113 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:45.706+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:46.113 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:45.706+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:46.113 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:45.706+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:46.113 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:45.706+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:46.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:46.347+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:46.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:46.347+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:46.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:46.347+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:46.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:46.347+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:47.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:46.918+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:47.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:46.918+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:47.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:46.918+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:47.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:46.919+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:47.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:46.715+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:47.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:46.715+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:47.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:46.715+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:47.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:46.715+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:47.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:47.313+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:47.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:47.314+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:47.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:47.314+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:47.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:47.314+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:48.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:05:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:05:47.725+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 12065 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:05:48.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:47.916+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:48.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:47.916+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:48.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:47.916+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:48.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:47.917+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:48.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:47.725+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:48.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:47.725+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:48.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:47.725+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:48.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:47.725+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:48.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:48.361+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:48.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:48.361+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:48.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:48.361+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:48.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:48.361+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:49.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:48.888+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:49.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:48.888+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:49.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:48.888+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:49.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:48.888+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:49.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:48.709+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:49.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:48.709+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:49.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:48.709+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:49.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:48.710+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:49.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:49.378+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:49.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:49.378+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:49.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:49.378+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:49.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:49.378+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:50.132 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:49.738+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:50.132 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:49.738+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:50.132 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:49.738+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:50.132 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:49.739+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:50.133 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:49.894+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:50.133 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:49.894+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:50.133 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:49.894+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:50.133 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:49.895+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:50.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:50.394+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:50.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:50.394+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:50.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:50.394+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:50.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:50.394+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:51.162 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:50.708+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:51.162 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:50.708+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:51.162 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:50.708+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:51.162 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:50.708+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:51.163 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:50.862+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:51.163 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:50.862+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:51.163 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:50.862+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:51.163 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:50.862+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:51.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:51.425+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:51.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:51.425+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:51.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:51.425+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:51.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:51.425+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:52.000 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:51.732+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:52.000 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:51.732+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:52.000 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:51.732+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:52.000 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:51.732+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:52.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:51.907+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:52.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:51.907+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:52.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:51.907+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:52.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:51.908+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:52.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:52.407+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:52.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:52.407+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:52.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:52.407+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:52.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:52.407+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:53.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:05:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:05:52.728+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 12158 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:05:53.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:52.717+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:53.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:52.717+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:53.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:52.717+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:53.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:52.718+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:53.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:52.885+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:53.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:52.885+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:53.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:52.885+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:53.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:52.886+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:53.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:53.396+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:53.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:53.396+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:53.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:53.396+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:53.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:53.396+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:54.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:53.718+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:54.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:53.718+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:54.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:53.718+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:54.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:53.719+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:54.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:53.918+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:54.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:53.918+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:54.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:53.918+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:54.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:53.919+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:54.745 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:54.442+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:54.745 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:54.442+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:54.745 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:54.442+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:54.745 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:54.442+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:55.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:54.759+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:55.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:54.759+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:55.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:54.759+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:55.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:54.760+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:55.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:54.910+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:55.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:54.910+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:55.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:54.910+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:55.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:54.911+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:55.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:55.438+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:55.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:55.438+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:55.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:55.438+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:55.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:55.438+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:56.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:55.728+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:56.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:55.728+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:56.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:55.728+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:56.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:55.728+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:56.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:55.940+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:56.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:55.940+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:56.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:55.940+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:56.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:55.940+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:56.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:56.455+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:56.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:56.455+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:56.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:56.455+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:56.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:56.456+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:57.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:56.758+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:57.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:56.758+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:57.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:56.758+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:57.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:56.758+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:57.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:56.989+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:57.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:56.989+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:57.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:56.989+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:57.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:56.989+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:57.715 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:57.470+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:57.716 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:57.470+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:57.716 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:57.470+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:57.716 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:57.470+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:58.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:05:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:05:57.730+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 12255 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:05:58.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:57.749+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:58.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:57.749+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:58.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:57.749+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:58.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:57.750+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:58.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:57.960+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:58.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:57.960+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:58.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:57.960+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:58.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:57.960+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:58.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:58.424+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:58.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:58.424+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:58.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:58.424+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:58.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:58.424+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:59.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:58.947+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:59.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:58.947+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:59.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:58.947+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:05:59.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:58.947+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:05:59.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:58.737+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:59.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:58.737+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:59.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:58.737+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:05:59.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:58.737+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:05:59.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:59.383+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:59.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:59.383+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:59.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:59.383+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:05:59.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:05:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:05:59.383+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:00.133 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:59.932+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:00.133 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:59.932+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:00.134 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:59.932+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:00.134 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:05:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:05:59.933+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:00.134 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:59.787+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:00.134 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:59.787+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:00.134 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:59.787+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:00.134 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:05:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:05:59.787+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:00.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:00.343+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:00.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:00.343+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:00.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:00.343+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:00.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:00.343+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:01.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:00.919+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:01.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:00.919+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:01.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:00.919+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:01.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:00.919+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:01.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:00.758+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:01.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:00.758+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:01.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:00.758+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:01.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:00.758+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:01.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:01.324+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:01.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:01.324+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:01.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:01.324+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:01.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:01.325+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:02.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:01.957+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:02.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:01.957+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:02.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:01.957+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:02.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:01.957+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:02.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:01.805+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:02.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:01.805+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:02.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:01.805+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:02.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:01.805+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:02.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:02.275+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:02.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:02.275+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:02.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:02.275+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:02.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:02.275+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:03.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:06:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:06:02.732+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 12349 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:06:03.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:02.954+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:03.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:02.954+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:03.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:02.954+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:03.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:02.954+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:03.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:02.780+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:03.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:02.780+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:03.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:02.780+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:03.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:02.780+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:03.566 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:03.256+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:03.566 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:03.256+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:03.566 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:03.256+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:03.566 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:03.256+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:03.819 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:03.812+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:03.819 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:03.812+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:03.819 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:03.812+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:03.819 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:03.813+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:04.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:03.911+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:04.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:03.911+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:04.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:03.911+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:04.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:03.911+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:04.582 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:04.260+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:04.582 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:04.260+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:04.582 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:04.260+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:04.583 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:04.260+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:05.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:04.924+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:05.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:04.924+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:05.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:04.924+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:05.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:04.925+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:05.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:04.818+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:05.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:04.818+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:05.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:04.818+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:05.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:04.818+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:05.541 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:05.277+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:05.542 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:05.277+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:05.542 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:05.277+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:05.542 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:05.277+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:06.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:05.839+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:06.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:05.839+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:06.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:05.839+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:06.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:05.839+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:06.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:05.876+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:06.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:05.876+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:06.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:05.876+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:06.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:05.876+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:06.630 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:06.240+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:06.630 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:06.240+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:06.631 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:06.240+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:06.631 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:06.241+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:07.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:06.900+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:07.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:06.900+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:07.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:06.900+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:07.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:06.901+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:07.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:06.860+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:07.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:06.860+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:07.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:06.860+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:07.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:06.860+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:07.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:07.199+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:07.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:07.199+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:07.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:07.199+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:07.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:07.200+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:08.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:06:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:06:07.734+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 12444 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:06:08.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:07.898+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:08.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:07.898+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:08.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:07.898+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:08.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:07.898+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:08.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:07.908+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:08.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:07.908+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:08.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:07.908+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:08.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:07.908+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:08.614 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:08.210+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:08.614 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:08.210+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:08.614 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:08.210+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:08.614 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:08.210+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:09.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:08.950+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:09.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:08.950+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:09.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:08.950+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:09.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:08.951+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:09.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:08.878+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:09.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:08.878+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:09.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:08.878+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:09.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:08.878+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:09.674 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:09.260+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:09.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:09.260+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:09.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:09.260+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:09.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:09.260+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:10.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:09.915+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:10.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:09.915+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:10.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:09.915+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:10.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:09.915+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:10.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:09.920+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:10.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:09.920+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:10.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:09.920+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:10.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:09.921+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:10.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:10.231+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:10.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:10.231+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:10.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:10.231+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:10.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:10.232+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:11.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:10.877+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:11.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:10.877+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:11.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:10.877+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:11.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:10.877+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:11.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:10.910+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:11.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:10.910+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:11.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:10.910+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:11.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:10.911+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:11.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:11.258+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:11.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:11.258+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:11.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:11.258+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:11.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:11.258+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:12.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:11.913+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:12.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:11.913+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:12.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:11.913+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:12.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:11.914+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:12.001 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:11.938+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:12.001 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:11.938+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:12.002 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:11.938+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:12.002 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:11.939+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:12.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:12.223+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:12.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:12.223+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:12.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:12.223+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:12.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:12.223+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:12.846 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:06:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:06:12.736+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 12538 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:06:13.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:12.923+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:13.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:12.924+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:13.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:12.924+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:13.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:12.924+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:13.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:12.916+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:13.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:12.916+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:13.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:12.916+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:13.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:12.917+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:13.567 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:13.235+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:13.567 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:13.235+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:13.567 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:13.235+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:13.568 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:13.235+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:14.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:13.939+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:14.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:13.939+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:14.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:13.939+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:14.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:13.940+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:14.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:13.881+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:14.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:13.881+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:14.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:13.881+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:14.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:13.881+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:14.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:14.265+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:14.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:14.265+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:14.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:14.265+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:14.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:14.266+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:15.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:14.952+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:15.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:14.952+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:15.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:14.952+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:15.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:14.952+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:15.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:14.854+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:15.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:14.854+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:15.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:14.854+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:15.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:14.854+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:15.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:15.249+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:15.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:15.249+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:15.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:15.249+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:15.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:15.249+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:16.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:15.927+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:16.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:15.927+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:16.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:15.927+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:16.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:15.927+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:16.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:15.852+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:16.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:15.852+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:16.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:15.852+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:16.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:15.852+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:16.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:16.210+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:16.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:16.210+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:16.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:16.210+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:16.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:16.210+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:17.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:16.889+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:17.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:16.889+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:17.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:16.889+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:17.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:16.889+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:17.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:16.896+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:17.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:16.896+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:17.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:16.896+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:17.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:16.897+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:17.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:17.213+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:17.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:17.213+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:17.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:17.213+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:17.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:17.213+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:18.161 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:06:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:06:17.738+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 12634 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:06:18.162 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:17.845+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:18.162 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:17.845+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:18.162 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:17.845+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:18.162 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:17.846+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:18.162 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:17.863+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:18.162 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:17.863+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:18.162 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:17.863+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:18.163 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:17.863+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:18.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:18.176+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:18.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:18.176+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:18.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:18.176+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:18.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:18.176+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:19.150 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:18.868+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:19.150 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:18.868+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:19.151 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:18.868+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:19.151 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:18.869+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:19.151 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:18.853+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:19.151 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:18.853+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:19.151 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:18.853+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:19.151 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:18.853+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:19.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:19.165+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:19.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:19.165+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:19.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:19.165+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:19.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:19.165+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:20.135 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:19.855+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:20.135 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:19.855+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:20.135 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:19.855+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:20.135 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:19.856+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:20.135 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:19.846+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:20.135 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:19.846+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:20.135 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:19.846+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:20.135 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:19.846+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:20.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:20.158+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:20.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:20.158+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:20.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:20.158+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:20.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:20.158+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:21.133 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:20.850+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:21.133 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:20.850+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:21.133 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:20.850+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:21.133 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:20.851+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:21.133 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:20.797+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:21.133 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:20.797+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:21.134 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:20.797+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:21.134 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:20.797+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:21.424 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:21.148+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:21.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:21.148+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:21.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:21.148+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:21.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:21.148+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:22.108 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:21.835+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:22.108 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:21.835+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:22.108 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:21.835+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:22.108 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:21.835+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:22.108 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:21.768+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:22.108 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:21.768+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:22.108 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:21.768+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:22.108 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:21.769+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:22.424 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:22.122+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:22.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:22.122+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:22.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:22.122+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:22.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:22.122+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:23.092 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:06:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:06:22.741+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 12729 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:06:23.092 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:22.814+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:23.092 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:22.814+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:23.092 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:22.814+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:23.092 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:22.815+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:23.092 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:22.818+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:23.093 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:22.818+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:23.093 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:22.818+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:23.093 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:22.818+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:23.225 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-06T03:06:23.238+0000 7f4f53757700 0 monclient(hunting): authenticate timed out after 300 2024-04-06T03:06:23.225 INFO:teuthology.orchestra.run.smithi105.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-04-06T03:06:23.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:23.106+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:23.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:23.106+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:23.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:23.106+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:23.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:23.106+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:23.840 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:23.788+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:23.840 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:23.788+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:23.841 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:23.788+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:23.841 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:23.788+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:24.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:06:24.076 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/cephadm.py", line 1364, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi105 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-04-06T03:06:24.296 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=14ac10fa5df845a0a9e4ccf6db7a87c5 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/cephadm.py", line 1364, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi105 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-04-06T03:06:24.300 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-06T03:06:24.312 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-06T03:06:24.312 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-06T03:06:24.313 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T03:06:24.318 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:23.855+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:24.318 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:23.855+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:24.319 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:23.855+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:24.319 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:23.855+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:24.339 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-06T03:06:24.340 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T03:06:24.400 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T03:06:24.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:24.136+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:24.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:24.136+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:24.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:24.136+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:24.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:24.136+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:25.155 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:24.827+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:25.156 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:24.827+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:25.156 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:24.827+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:25.156 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:24.828+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:25.156 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:24.752+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:25.156 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:24.752+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:25.156 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:24.752+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:25.157 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:24.752+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:25.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:25.170+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:25.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:25.170+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:25.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:25.170+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:25.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:25.170+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:26.162 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:25.747+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:26.162 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:25.747+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:26.163 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:25.747+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:26.163 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:25.747+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:26.163 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:25.821+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:26.163 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:25.821+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:26.163 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:25.821+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:26.163 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:25.821+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:26.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:26.177+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:26.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:26.177+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:26.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:26.177+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:26.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:26.177+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:27.046 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:26.792+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:27.046 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:26.792+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:27.046 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:26.792+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:27.047 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:26.793+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:27.047 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:26.840+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:27.047 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:26.840+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:27.047 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:26.840+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:27.047 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:26.840+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:27.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:27.194+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:27.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:27.194+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:27.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:27.194+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:27.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:27.194+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:28.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:06:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:06:27.743+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 12824 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:06:28.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:27.815+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:28.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:27.815+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:28.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:27.815+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:28.177 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:27.816+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:28.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:27.789+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:28.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:27.789+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:28.178 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:27.789+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:28.178 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:27.789+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:28.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:28.210+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:28.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:28.210+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:28.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:28.210+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:28.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:28.210+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:29.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:28.781+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:29.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:28.781+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:29.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:28.781+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:29.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:28.781+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:29.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:28.816+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:29.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:28.816+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:29.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:28.816+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:29.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:28.816+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:29.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:29.260+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:29.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:29.260+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:29.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:29.260+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:29.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:29.260+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:30.118 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:29.737+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:30.118 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:29.737+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:30.119 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:29.737+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:30.119 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:29.737+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:30.119 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:29.838+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:30.119 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:29.838+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:30.119 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:29.838+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:30.119 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:29.838+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:30.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:30.235+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:30.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:30.235+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:30.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:30.235+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:30.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:30.235+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:31.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:30.776+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:31.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:30.776+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:31.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:30.776+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:31.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:30.776+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:31.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:30.868+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:31.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:30.868+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:31.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:30.868+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:31.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:30.869+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:31.524 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:31.223+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:31.525 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:31.223+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:31.525 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:31.223+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:31.525 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:31.223+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:31.805 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:31.736+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:31.805 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:31.736+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:31.806 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:31.736+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:31.806 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:31.736+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:32.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:31.820+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:32.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:31.820+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:32.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:31.820+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:32.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:31.820+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:32.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:32.257+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:32.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:32.257+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:32.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:32.257+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:32.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:32.257+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:33.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:06:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:06:32.745+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 12918 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:06:33.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:32.698+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:33.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:32.698+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:33.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:32.698+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:33.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:32.698+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:33.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:32.851+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:33.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:32.851+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:33.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:32.851+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:33.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:32.851+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:33.568 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:33.244+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:33.568 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:33.244+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:33.568 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:33.244+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:33.568 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:33.244+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:33.834 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:33.679+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:33.834 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:33.679+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:33.835 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:33.679+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:33.835 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:33.680+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:34.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:33.848+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:34.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:33.848+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:34.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:33.848+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:34.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:33.848+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:34.640 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:34.262+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:34.641 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:34.262+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:34.641 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:34.262+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:34.641 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:34.262+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:34.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:34.655+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:34.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:34.655+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:34.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:34.655+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:34.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:34.655+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:34.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:34.887+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:34.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:34.887+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:34.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:34.887+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:34.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:34.887+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:35.604 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:35.265+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:35.605 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:35.265+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:35.605 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:35.265+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:35.605 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:35.265+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:35.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:35.618+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:35.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:35.618+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:35.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:35.618+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:35.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:35.618+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:35.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:35.863+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:35.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:35.863+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:35.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:35.863+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:35.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:35.863+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:36.567 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:36.251+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:36.567 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:36.251+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:36.567 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:36.251+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:36.567 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:36.251+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:36.846 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:36.608+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:36.847 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:36.608+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:36.847 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:36.608+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:36.847 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:36.608+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:37.135 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:36.861+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:37.135 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:36.861+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:37.135 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:36.861+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:37.136 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:36.861+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:37.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:37.228+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:37.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:37.228+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:37.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:37.228+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:37.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:37.228+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:37.854 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:06:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:06:37.747+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 13015 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:06:37.854 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:37.601+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:37.854 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:37.601+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:37.855 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:37.601+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:37.855 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:37.602+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:37.855 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:37.819+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:37.855 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:37.819+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:37.855 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:37.819+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:37.855 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:37.819+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:38.633 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:38.195+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:38.634 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:38.195+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:38.634 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:38.195+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:38.634 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:38.195+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:38.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:38.648+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:38.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:38.648+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:38.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:38.648+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:38.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:38.648+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:38.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:38.806+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:38.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:38.806+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:38.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:38.806+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:38.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:38.807+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:39.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:39.153+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:39.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:39.153+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:39.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:39.153+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:39.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:39.153+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:39.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:39.664+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:39.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:39.664+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:39.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:39.664+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:39.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:39.664+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:39.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:39.804+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:39.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:39.804+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:39.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:39.804+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:39.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:39.804+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:40.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:40.195+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:40.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:40.195+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:40.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:40.195+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:40.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:40.195+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:40.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:40.615+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:40.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:40.615+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:40.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:40.615+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:40.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:40.616+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:40.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:40.779+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:40.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:40.779+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:40.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:40.779+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:40.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:40.780+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:41.577 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:41.219+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:41.577 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:41.219+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:41.577 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:41.219+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:41.577 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:41.219+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:41.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:41.591+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:41.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:41.591+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:41.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:41.591+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:41.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:41.592+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:41.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:41.757+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:41.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:41.757+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:41.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:41.757+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:41.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:41.757+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:42.278 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:42.172+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:42.278 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:42.172+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:42.279 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:42.172+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:42.279 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:42.172+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:42.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:42.605+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:42.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:42.605+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:42.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:42.605+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:42.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:42.605+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:43.135 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:06:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:06:42.749+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 13106 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:06:43.135 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:42.755+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:43.136 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:42.755+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:43.136 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:42.755+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:43.136 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:42.756+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:43.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:43.149+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:43.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:43.149+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:43.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:43.149+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:43.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:43.150+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:43.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:43.584+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:43.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:43.584+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:43.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:43.584+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:43.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:43.585+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:43.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:43.796+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:43.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:43.796+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:43.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:43.796+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:43.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:43.797+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:44.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:44.125+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:44.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:44.125+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:44.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:44.125+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:44.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:44.125+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:44.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:44.550+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:44.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:44.550+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:44.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:44.550+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:44.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:44.550+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:44.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:44.747+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:44.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:44.747+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:44.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:44.747+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:44.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:44.748+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:45.424 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:45.088+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:45.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:45.088+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:45.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:45.088+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:45.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:45.088+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:45.774 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:45.517+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:45.774 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:45.517+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:45.774 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:45.517+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:45.774 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:45.518+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:46.069 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:45.788+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:46.069 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:45.788+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:46.069 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:45.788+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:46.069 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:45.789+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:46.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:46.083+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:46.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:46.083+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:46.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:46.083+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:46.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:46.084+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:46.797 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:46.493+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:46.797 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:46.493+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:46.798 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:46.493+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:46.798 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:46.493+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:47.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:46.812+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:47.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:46.812+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:47.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:46.812+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:47.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:46.812+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:47.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:47.039+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:47.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:47.039+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:47.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:47.039+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:47.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:47.040+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:47.737 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:47.497+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:47.737 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:47.497+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:47.737 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:47.497+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:47.738 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:47.497+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:47.998 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:06:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:06:47.751+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 13201 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:06:47.999 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:47.826+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:47.999 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:47.826+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:48.000 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:47.826+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:48.000 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:47.827+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:48.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:48.011+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:48.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:48.011+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:48.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:48.011+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:48.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:48.012+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:48.862 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:48.522+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:48.863 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:48.522+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:48.863 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:48.522+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:48.863 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:48.522+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:49.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:48.984+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:49.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:48.984+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:49.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:48.984+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:49.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:48.984+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:49.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:48.877+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:49.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:48.877+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:49.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:48.877+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:49.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:48.877+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:49.862 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:49.519+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:49.862 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:49.519+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:49.862 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:49.519+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:49.862 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:49.520+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:50.136 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:49.876+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:50.136 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:49.876+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:50.136 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:49.876+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:50.136 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:49.876+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:50.136 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:50.015+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:50.137 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:50.015+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:50.137 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:50.015+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:50.137 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:50.015+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:50.896 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:50.526+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:50.896 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:50.526+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:50.896 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:50.526+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:50.897 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:50.526+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:51.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:50.911+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:51.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:50.911+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:51.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:50.911+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:51.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:50.911+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:51.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:51.018+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:51.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:51.018+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:51.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:51.018+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:51.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:51.019+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:51.916 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:51.492+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:51.916 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:51.492+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:51.916 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:51.492+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:51.916 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:51.492+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:52.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:51.930+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:52.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:51.930+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:52.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:51.930+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:52.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:51.930+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:52.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:52.019+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:52.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:52.019+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:52.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:52.019+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:52.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:52.019+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:52.739 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:52.485+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:52.739 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:52.485+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:52.739 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:52.485+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:52.739 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:52.486+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:53.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:06:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:06:52.753+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 13296 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:06:53.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:52.959+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:53.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:52.959+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:53.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:52.959+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:53.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:52.959+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:53.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:52.977+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:53.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:52.977+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:53.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:52.977+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:53.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:52.977+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:53.909 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:53.502+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:53.909 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:53.502+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:53.909 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:53.502+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:53.910 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:53.502+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:54.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:53.923+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:54.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:53.923+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:54.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:53.923+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:54.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:53.924+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:54.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:53.942+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:54.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:53.942+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:54.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:53.942+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:54.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:53.942+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:54.878 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:54.472+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:54.879 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:54.472+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:54.879 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:54.472+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:54.879 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:54.472+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:55.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:54.893+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:55.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:54.893+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:55.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:54.893+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:55.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:54.893+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:55.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:54.963+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:55.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:54.963+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:55.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:54.963+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:55.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:54.963+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:55.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:55.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:55.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:55.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:55.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:55.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:55.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:55.500+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:56.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:55.943+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:56.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:55.943+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:56.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:55.943+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:56.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:55.943+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:56.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:55.955+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:56.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:55.955+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:56.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:55.955+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:56.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:55.956+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:56.733 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:56.478+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:56.733 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:56.478+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:56.733 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:56.478+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:56.734 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:56.478+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:57.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:56.979+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:57.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:56.979+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:57.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:56.979+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:57.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:56.980+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:57.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:56.943+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:57.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:56.943+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:57.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:56.943+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:57.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:56.943+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:57.741 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:57.455+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:57.742 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:57.455+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:57.742 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:57.455+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:57.742 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:57.455+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:58.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:06:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:06:57.756+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 13392 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:06:58.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:57.951+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:58.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:57.951+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:58.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:57.951+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:58.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:57.951+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:58.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:57.910+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:58.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:57.910+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:58.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:57.910+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:58.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:57.910+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:58.674 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:58.429+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:58.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:58.429+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:58.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:58.429+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:58.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:58.430+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:59.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:58.923+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:59.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:58.923+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:59.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:58.923+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:06:59.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:58.923+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:06:59.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:58.954+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:59.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:58.954+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:59.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:58.954+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:06:59.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:58.954+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:06:59.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:59.388+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:59.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:59.388+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:59.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:59.388+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:06:59.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:06:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:06:59.388+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:00.397 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:59.947+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:00.398 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:59.947+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:00.398 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:59.947+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:00.398 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:06:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:06:59.948+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:00.398 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:59.997+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:00.399 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:59.997+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:00.399 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:59.997+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:00.399 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:06:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:06:59.997+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:00.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:00.412+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:00.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:00.412+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:00.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:00.412+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:00.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:00.412+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:01.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:00.980+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:01.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:00.980+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:01.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:00.980+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:01.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:00.980+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:01.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:00.962+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:01.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:00.962+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:01.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:00.962+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:01.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:00.962+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:01.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:01.429+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:01.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:01.429+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:01.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:01.429+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:01.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:01.430+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:02.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:02.028+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:02.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:02.028+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:02.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:02.028+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:02.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:02.029+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:02.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:02.012+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:02.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:02.012+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:02.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:02.012+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:02.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:02.012+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:02.744 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:02.447+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:02.746 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:02.447+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:02.746 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:02.447+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:02.746 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:02.447+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:03.012 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:07:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:07:02.758+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 13486 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:07:03.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:03.027+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:03.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:03.027+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:03.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:03.027+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:03.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:03.027+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:03.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:03.079+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:03.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:03.079+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:03.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:03.079+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:03.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:03.079+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:03.909 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:03.469+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:03.910 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:03.469+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:03.910 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:03.469+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:03.910 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:03.469+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:04.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:04.034+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:04.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:04.034+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:04.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:04.034+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:04.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:04.034+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:04.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:04.095+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:04.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:04.095+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:04.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:04.095+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:04.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:04.095+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:04.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:04.502+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:04.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:04.502+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:04.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:04.502+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:04.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:04.502+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:05.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:05.061+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:05.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:05.062+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:05.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:05.062+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:05.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:05.062+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:05.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:05.049+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:05.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:05.049+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:05.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:05.049+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:05.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:05.049+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:05.924 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:05.521+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:05.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:05.521+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:05.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:05.521+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:05.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:05.521+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:06.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:06.069+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:06.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:06.069+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:06.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:06.069+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:06.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:06.069+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:06.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:06.014+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:06.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:06.014+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:06.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:06.014+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:06.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:06.014+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:06.819 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:06.554+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:06.819 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:06.554+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:06.819 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:06.554+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:06.819 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:06.554+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:07.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:06.978+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:07.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:06.978+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:07.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:06.978+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:07.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:06.978+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:07.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:07.064+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:07.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:07.064+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:07.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:07.064+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:07.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:07.065+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:07.870 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:07:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:07:07.760+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 13580 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:07:07.871 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:07.581+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:07.871 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:07.581+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:07.871 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:07.581+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:07.871 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:07.582+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:08.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:08.020+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:08.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:08.020+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:08.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:08.020+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:08.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:08.021+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:08.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:07.972+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:08.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:07.972+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:08.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:07.972+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:08.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:07.972+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:08.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:08.537+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:08.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:08.537+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:08.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:08.537+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:08.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:08.537+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:09.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:09.036+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:09.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:09.036+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:09.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:09.036+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:09.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:09.036+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:09.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:09.019+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:09.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:09.019+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:09.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:09.019+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:09.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:09.019+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:09.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:09.542+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:09.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:09.542+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:09.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:09.542+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:09.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:09.542+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:10.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:10.076+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:10.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:10.076+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:10.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:10.076+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:10.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:10.076+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:10.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:09.991+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:10.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:09.991+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:10.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:09.991+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:10.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:09.991+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:10.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:10.528+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:10.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:10.528+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:10.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:10.528+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:10.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:10.529+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:11.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:11.103+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:11.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:11.103+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:11.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:11.103+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:11.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:11.103+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:11.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:10.994+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:11.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:10.994+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:11.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:10.994+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:11.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:10.994+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:11.862 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:11.566+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:11.862 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:11.566+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:11.862 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:11.566+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:11.862 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:11.567+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:12.126 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:11.951+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:12.126 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:11.951+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:12.127 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:11.951+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:12.127 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:11.951+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:12.424 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:12.141+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:12.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:12.141+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:12.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:12.141+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:12.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:12.141+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:12.872 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:07:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:07:12.763+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 13673 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:07:12.873 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:12.612+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:12.873 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:12.612+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:12.873 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:12.612+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:12.873 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:12.612+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:13.126 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:12.933+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:13.126 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:12.933+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:13.126 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:12.933+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:13.126 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:12.933+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:13.127 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:13.121+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:13.127 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:13.121+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:13.127 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:13.121+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:13.127 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:13.121+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:13.888 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:13.660+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:13.888 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:13.660+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:13.888 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:13.660+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:13.889 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:13.660+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:14.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:13.903+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:14.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:13.903+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:14.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:13.903+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:14.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:13.903+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:14.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:14.108+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:14.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:14.108+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:14.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:14.108+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:14.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:14.108+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:14.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:14.697+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:14.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:14.697+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:14.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:14.697+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:14.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:14.697+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:15.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:14.949+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:15.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:14.949+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:15.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:14.949+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:15.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:14.949+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:15.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:15.068+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:15.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:15.068+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:15.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:15.068+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:15.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:15.068+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:16.029 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:15.701+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:16.029 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:15.701+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:16.030 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:15.701+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:16.030 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:15.701+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:16.030 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:15.929+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:16.031 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:15.929+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:16.031 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:15.929+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:16.031 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:15.929+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:16.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:16.044+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:16.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:16.044+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:16.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:16.044+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:16.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:16.044+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:16.958 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:16.698+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:16.958 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:16.698+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:16.959 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:16.698+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:16.959 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:16.698+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:17.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:16.972+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:17.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:16.972+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:17.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:16.973+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:17.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:16.973+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:17.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:17.075+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:17.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:17.075+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:17.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:17.075+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:17.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:17.075+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:17.950 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:07:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:07:17.765+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 13768 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:07:17.951 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:17.710+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:17.951 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:17.710+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:17.951 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:17.710+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:17.952 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:17.710+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:18.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:17.966+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:18.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:17.966+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:18.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:17.966+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:18.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:17.966+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:18.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:18.096+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:18.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:18.096+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:18.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:18.096+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:18.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:18.096+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:19.127 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:18.944+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:19.128 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:18.944+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:19.128 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:18.944+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:19.128 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:18.944+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:19.128 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:18.748+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:19.129 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:18.748+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:19.129 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:18.748+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:19.129 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:18.748+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:19.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:19.142+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:19.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:19.142+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:19.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:19.142+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:19.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:19.142+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:20.137 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:19.953+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:20.137 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:19.953+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:20.138 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:19.953+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:20.138 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:19.953+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:20.138 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:19.758+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:20.138 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:19.758+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:20.138 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:19.758+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:20.138 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:19.758+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:20.424 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:20.156+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:20.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:20.156+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:20.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:20.156+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:20.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:20.156+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:21.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:20.976+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:21.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:20.976+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:21.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:20.976+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:21.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:20.976+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:21.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:20.742+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:21.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:20.742+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:21.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:20.742+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:21.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:20.742+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:21.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:21.206+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:21.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:21.206+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:21.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:21.206+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:21.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:21.206+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:21.986 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:21.740+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:21.987 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:21.740+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:21.987 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:21.740+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:21.987 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:21.740+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:22.341 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:22.000+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:22.342 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:22.000+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:22.342 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:22.000+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:22.342 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:22.000+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:22.342 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:22.230+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:22.342 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:22.230+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:22.343 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:22.230+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:22.343 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:22.230+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:23.010 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:07:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:07:22.766+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 13862 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:07:23.010 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:22.724+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:23.011 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:22.724+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:23.011 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:22.724+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:23.011 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:22.724+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:23.424 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:23.255+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:23.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:23.255+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:23.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:23.255+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:23.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:23.255+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:23.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:23.023+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:23.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:23.023+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:23.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:23.023+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:23.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:23.023+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:23.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:23.717+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:23.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:23.717+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:23.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:23.717+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:23.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:23.717+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:24.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:24.217+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:24.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:24.217+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:24.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:24.217+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:24.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:24.217+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:24.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:24.016+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:24.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:24.016+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:24.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:24.016+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:24.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:24.016+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:24.984 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:24.731+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:24.985 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:24.731+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:24.985 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:24.731+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:24.985 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:24.731+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:25.414 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:25.220+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:25.414 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:25.220+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:25.414 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:25.220+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:25.414 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:25.220+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:25.415 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:24.999+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:25.415 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:24.999+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:25.415 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:24.999+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:25.415 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:24.999+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:25.924 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:25.684+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:25.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:25.684+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:25.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:25.684+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:25.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:25.684+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:26.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:26.213+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:26.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:26.213+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:26.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:26.213+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:26.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:26.213+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:26.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:26.032+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:26.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:26.032+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:26.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:26.032+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:26.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:26.032+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:26.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:26.638+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:26.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:26.638+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:26.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:26.638+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:26.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:26.638+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:27.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:27.173+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:27.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:27.173+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:27.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:27.173+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:27.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:27.174+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:27.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:26.999+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:27.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:26.999+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:27.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:26.999+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:27.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:27.000+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:27.881 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:07:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:07:27.770+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 13956 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:07:27.882 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:27.620+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:27.882 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:27.620+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:27.882 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:27.620+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:27.882 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:27.620+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:28.140 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:27.982+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:28.140 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:27.982+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:28.140 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:27.982+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:28.140 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:27.982+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:28.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:28.154+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:28.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:28.154+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:28.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:28.154+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:28.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:28.155+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:28.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:28.646+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:28.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:28.646+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:28.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:28.646+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:28.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:28.646+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:29.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:29.173+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:29.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:29.173+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:29.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:29.173+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:29.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:29.173+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:29.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:28.946+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:29.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:28.946+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:29.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:28.946+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:29.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:28.946+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:29.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:29.683+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:29.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:29.683+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:29.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:29.683+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:29.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:29.683+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:30.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:30.172+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:30.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:30.172+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:30.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:30.172+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:30.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:30.173+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:30.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:29.964+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:30.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:29.964+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:30.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:29.964+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:30.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:29.965+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:30.981 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:30.732+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:30.981 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:30.732+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:30.981 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:30.732+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:30.982 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:30.733+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:31.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:30.995+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:31.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:30.995+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:31.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:30.995+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:31.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:30.996+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:31.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:31.217+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:31.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:31.217+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:31.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:31.217+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:31.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:31.218+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:31.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:31.757+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:31.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:31.757+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:31.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:31.757+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:31.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:31.758+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:32.230 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:31.974+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:32.230 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:31.974+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:32.230 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:31.974+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:32.230 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:31.974+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:32.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:32.244+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:32.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:32.244+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:32.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:32.244+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:32.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:32.244+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:33.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:07:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:07:32.772+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 14050 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:07:33.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:32.951+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:33.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:32.951+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:33.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:32.951+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:33.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:32.951+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:33.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:32.780+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:33.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:32.780+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:33.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:32.780+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:33.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:32.780+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:33.572 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:33.212+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:33.572 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:33.212+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:33.572 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:33.212+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:33.572 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:33.213+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:33.907 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:33.777+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:33.907 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:33.777+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:33.907 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:33.777+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:33.908 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:33.777+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:34.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:33.922+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:34.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:33.922+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:34.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:33.922+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:34.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:33.922+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:34.629 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:34.222+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:34.630 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:34.222+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:34.630 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:34.222+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:34.630 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:34.222+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:34.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:34.797+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:34.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:34.797+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:34.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:34.797+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:34.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:34.797+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:35.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:35.206+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:35.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:35.206+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:35.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:35.206+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:35.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:35.206+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:35.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:34.969+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:35.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:34.969+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:35.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:34.969+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:35.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:34.970+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:36.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:35.766+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:36.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:35.766+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:36.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:35.766+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:36.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:35.766+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:36.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:36.011+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:36.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:36.011+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:36.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:36.011+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:36.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:36.011+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:36.674 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:36.244+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:36.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:36.244+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:36.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:36.244+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:36.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:36.245+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:37.074 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:37.023+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:37.075 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:37.023+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:37.075 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:37.023+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:37.075 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:37.023+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:37.075 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:36.810+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:37.075 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:36.810+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:37.075 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:36.810+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:37.076 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:36.811+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:37.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:37.242+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:37.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:37.242+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:37.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:37.242+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:37.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:37.243+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:37.884 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:07:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:07:37.774+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 14146 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:07:37.885 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:37.786+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:37.885 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:37.786+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:37.885 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:37.786+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:37.885 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:37.786+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:38.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:38.004+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:38.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:38.004+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:38.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:38.004+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:38.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:38.004+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:38.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:38.257+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:38.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:38.257+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:38.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:38.257+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:38.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:38.257+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:39.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:38.757+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:39.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:38.757+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:39.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:38.757+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:39.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:38.757+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:39.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:38.964+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:39.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:38.964+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:39.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:38.964+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:39.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:38.964+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:39.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:39.277+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:39.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:39.277+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:39.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:39.277+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:39.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:39.277+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:40.139 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:39.727+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:40.139 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:39.727+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:40.139 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:39.727+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:40.139 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:39.728+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:40.139 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:39.959+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:40.139 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:39.959+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:40.140 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:39.959+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:40.140 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:39.960+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:40.424 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:40.260+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:40.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:40.260+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:40.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:40.260+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:40.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:40.260+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:41.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:40.746+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:41.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:40.746+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:41.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:40.746+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:41.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:40.747+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:41.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:40.963+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:41.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:40.963+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:41.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:40.963+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:41.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:40.964+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:41.674 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:41.251+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:41.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:41.251+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:41.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:41.251+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:41.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:41.251+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:42.000 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:41.721+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:42.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:41.721+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:42.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:41.721+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:42.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:41.721+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:42.001 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:41.931+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:42.001 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:41.931+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:42.002 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:41.931+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:42.002 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:41.931+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:42.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:42.293+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:42.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:42.293+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:42.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:42.293+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:42.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:42.294+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:43.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:07:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:07:42.777+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 14240 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:07:43.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:42.752+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:43.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:42.753+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:43.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:42.753+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:43.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:42.753+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:43.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:42.893+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:43.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:42.893+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:43.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:42.893+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:43.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:42.894+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:43.572 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:43.266+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:43.572 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:43.266+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:43.572 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:43.266+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:43.573 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:43.267+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:43.845 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:43.708+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:43.846 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:43.708+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:43.846 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:43.708+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:43.846 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:43.708+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:44.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:43.879+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:44.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:43.879+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:44.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:43.879+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:44.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:43.880+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:44.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:44.313+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:44.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:44.313+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:44.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:44.313+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:44.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:44.313+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:45.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:44.708+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:45.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:44.708+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:45.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:44.708+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:45.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:44.708+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:45.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:44.926+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:45.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:44.927+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:45.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:44.927+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:45.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:44.927+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:45.674 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:45.363+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:45.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:45.363+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:45.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:45.363+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:45.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:45.363+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:46.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:45.732+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:46.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:45.732+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:46.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:45.732+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:46.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:45.732+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:46.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:45.893+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:46.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:45.893+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:46.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:45.893+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:46.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:45.894+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:46.674 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:46.370+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:46.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:46.370+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:46.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:46.370+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:46.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:46.370+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:47.160 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:46.691+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:47.160 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:46.691+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:47.160 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:46.691+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:47.160 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:46.691+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:47.161 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:46.846+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:47.161 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:46.846+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:47.161 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:46.846+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:47.161 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:46.846+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:47.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:47.386+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:47.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:47.386+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:47.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:47.386+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:47.425 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:47.387+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:48.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:07:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:07:47.779+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 14338 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:07:48.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:47.689+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:48.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:47.689+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:48.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:47.689+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:48.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:47.690+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:48.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:47.855+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:48.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:47.855+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:48.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:47.855+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:48.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:47.855+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:48.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:48.433+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:48.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:48.433+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:48.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:48.433+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:48.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:48.434+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:49.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:48.707+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:49.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:48.707+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:49.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:48.707+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:49.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:48.707+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:49.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:48.871+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:49.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:48.871+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:49.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:48.871+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:49.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:48.871+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:49.863 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:49.696+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:49.863 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:49.696+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:49.864 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:49.696+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:49.864 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:49.696+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:49.864 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:49.464+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:49.864 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:49.464+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:49.864 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:49.464+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:49.864 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:49.464+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:50.139 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:49.878+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:50.140 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:49.878+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:50.140 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:49.878+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:50.140 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:49.878+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:50.884 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:50.689+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:50.885 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:50.689+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:50.885 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:50.689+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:50.885 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:50.689+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:50.886 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:50.509+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:50.886 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:50.509+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:50.886 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:50.509+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:50.887 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:50.509+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:51.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:50.898+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:51.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:50.898+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:51.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:50.898+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:51.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:50.899+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:51.898 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:51.738+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:51.899 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:51.738+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:51.899 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:51.738+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:51.899 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:51.738+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:51.899 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:51.539+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:51.900 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:51.539+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:51.900 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:51.539+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:51.900 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:51.539+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:52.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:51.913+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:52.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:51.913+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:52.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:51.913+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:52.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:51.913+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:52.789 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:07:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:07:52.781+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 14433 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:07:52.789 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:52.717+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:52.789 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:52.717+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:52.789 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:52.717+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:52.789 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:52.717+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:52.790 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:52.549+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:52.790 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:52.549+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:52.790 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:52.549+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:52.790 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:52.549+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:53.061 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:52.896+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:53.062 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:52.896+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:53.062 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:52.896+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:53.062 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:52.896+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:53.911 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:53.594+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:53.911 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:53.594+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:53.912 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:53.594+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:53.912 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:53.595+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:53.912 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:53.708+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:53.912 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:53.708+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:53.912 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:53.708+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:53.912 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:53.708+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:54.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:53.926+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:54.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:53.926+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:54.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:53.926+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:54.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:53.926+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:54.924 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:54.747+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:54.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:54.747+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:54.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:54.747+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:54.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:54.748+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:54.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:54.572+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:54.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:54.572+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:54.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:54.572+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:54.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:54.572+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:55.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:54.939+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:55.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:54.939+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:55.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:54.939+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:55.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:54.939+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:55.897 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:55.754+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:55.898 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:55.754+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:55.898 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:55.754+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:55.898 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:55.754+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:55.898 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:55.572+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:55.899 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:55.572+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:55.899 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:55.572+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:55.899 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:55.572+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:56.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:55.912+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:56.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:55.912+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:56.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:55.912+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:56.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:55.912+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:56.891 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:56.752+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:56.891 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:56.752+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:56.892 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:56.752+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:56.892 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:56.752+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:56.892 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:56.608+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:56.892 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:56.608+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:56.892 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:56.608+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:56.892 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:56.608+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:57.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:56.905+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:57.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:56.906+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:57.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:56.906+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:57.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:56.906+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:57.881 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:07:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:07:57.784+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 14529 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:07:57.882 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:57.583+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:57.882 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:57.583+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:57.882 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:57.583+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:57.882 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:57.583+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:57.883 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:57.758+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:57.883 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:57.758+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:57.883 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:57.758+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:57.883 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:57.758+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:58.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:57.897+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:58.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:57.897+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:58.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:57.897+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:58.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:57.897+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:58.844 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:58.763+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:58.844 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:58.763+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:58.844 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:58.763+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:58.845 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:58.763+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:58.845 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:58.602+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:58.845 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:58.602+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:58.845 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:58.602+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:58.845 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:58.602+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:59.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:58.859+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:59.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:58.859+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:59.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:58.859+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:59.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:58.859+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:59.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:59.861+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:59.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:59.861+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:59.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:59.861+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:07:59.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:07:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:07:59.861+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:07:59.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:59.613+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:59.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:59.613+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:59.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:59.613+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:07:59.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:07:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:07:59.613+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:07:59.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:59.739+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:59.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:59.739+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:59.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:59.739+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:07:59.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:07:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:07:59.739+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:00.877 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:00.722+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:00.878 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:00.722+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:00.878 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:00.722+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:00.878 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:00.722+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:00.878 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:00.639+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:00.878 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:00.639+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:00.878 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:00.639+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:00.879 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:00.639+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:01.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:00.892+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:01.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:00.892+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:01.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:00.892+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:01.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:00.892+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:01.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:01.705+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:01.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:01.705+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:01.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:01.705+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:01.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:01.705+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:01.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:01.897+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:01.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:01.897+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:01.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:01.897+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:01.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:01.898+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:01.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:01.656+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:01.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:01.656+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:01.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:01.656+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:01.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:01.657+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:02.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:08:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:08:02.786+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 14625 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:08:02.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:02.677+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:02.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:02.677+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:02.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:02.677+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:02.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:02.677+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:02.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:02.755+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:02.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:02.755+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:02.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:02.755+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:02.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:02.755+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:02.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:02.907+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:02.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:02.907+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:02.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:02.907+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:02.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:02.907+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:03.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:03.710+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:03.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:03.710+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:03.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:03.710+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:03.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:03.711+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:03.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:03.678+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:03.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:03.678+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:03.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:03.678+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:03.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:03.679+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:04.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:03.953+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:04.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:03.953+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:04.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:03.953+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:04.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:03.953+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:04.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:04.666+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:04.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:04.666+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:04.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:04.666+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:04.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:04.666+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:04.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:04.638+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:04.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:04.638+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:04.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:04.638+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:04.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:04.638+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:05.349 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:04.988+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:05.350 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:04.988+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:05.350 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:04.988+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:05.350 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:04.988+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:05.655 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:05.596+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:05.656 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:05.596+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:05.656 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:05.596+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:05.656 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:05.596+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:05.924 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:05.670+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:05.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:05.670+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:05.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:05.670+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:05.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:05.671+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:06.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:05.995+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:06.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:05.995+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:06.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:05.995+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:06.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:05.995+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:06.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:06.694+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:06.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:06.694+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:06.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:06.694+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:06.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:06.695+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:06.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:06.615+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:06.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:06.615+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:06.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:06.615+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:06.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:06.615+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:07.329 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:06.976+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:07.329 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:06.976+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:07.330 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:06.976+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:07.330 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:06.976+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:07.899 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:08:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:08:07.789+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 14716 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:08:07.900 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:07.656+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:07.900 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:07.656+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:07.900 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:07.656+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:07.900 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:07.656+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:07.901 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:07.740+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:07.901 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:07.740+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:07.901 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:07.740+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:07.901 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:07.740+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:08.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:07.990+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:08.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:07.990+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:08.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:07.990+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:08.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:07.990+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:08.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:08.769+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:08.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:08.769+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:08.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:08.769+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:08.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:08.769+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:08.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:08.657+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:08.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:08.657+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:08.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:08.657+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:08.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:08.657+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:09.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:08.943+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:09.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:08.943+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:09.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:08.943+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:09.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:08.943+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:09.879 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:09.614+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:09.879 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:09.614+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:09.879 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:09.614+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:09.880 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:09.614+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:09.880 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:09.737+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:09.880 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:09.737+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:09.880 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:09.737+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:09.880 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:09.737+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:10.144 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:09.894+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:10.144 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:09.894+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:10.144 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:09.894+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:10.144 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:09.894+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:10.902 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:10.582+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:10.903 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:10.582+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:10.903 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:10.582+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:10.903 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:10.582+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:10.903 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:10.721+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:10.903 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:10.721+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:10.903 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:10.721+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:10.904 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:10.721+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:11.178 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:10.918+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:11.178 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:10.918+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:11.178 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:10.918+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:11.178 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:10.918+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:11.907 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:11.629+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:11.908 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:11.629+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:11.908 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:11.629+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:11.908 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:11.629+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:11.908 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:11.735+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:11.908 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:11.735+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:11.909 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:11.735+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:11.909 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:11.735+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:12.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:11.922+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:12.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:11.922+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:12.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:11.922+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:12.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:11.922+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:12.902 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:08:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:08:12.790+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 14806 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:08:12.903 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:12.653+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:12.903 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:12.653+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:12.903 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:12.653+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:12.903 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:12.653+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:12.903 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:12.754+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:12.903 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:12.754+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:12.904 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:12.754+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:12.904 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:12.754+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:12.904 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:12.892+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:12.904 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:12.892+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:12.904 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:12.892+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:12.904 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:12.892+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:13.831 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:13.785+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:13.831 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:13.785+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:13.832 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:13.785+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:13.832 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:13.785+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:13.832 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:13.649+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:13.832 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:13.649+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:13.832 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:13.649+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:13.832 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:13.649+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:14.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:13.846+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:14.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:13.846+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:14.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:13.846+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:14.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:13.846+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:14.820 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:14.669+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:14.821 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:14.669+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:14.821 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:14.669+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:14.821 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:14.669+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:15.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:14.835+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:15.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:14.835+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:15.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:14.835+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:15.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:14.835+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:15.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:14.843+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:15.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:14.843+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:15.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:14.843+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:15.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:14.844+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:16.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:15.846+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:16.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:15.846+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:16.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:15.846+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:16.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:15.847+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:16.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:15.705+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:16.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:15.706+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:16.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:15.706+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:16.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:15.706+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:16.177 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:15.807+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:16.177 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:15.807+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:16.177 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:15.807+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:16.178 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:15.808+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:17.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:16.853+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:17.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:16.853+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:17.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:16.853+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:17.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:16.853+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:17.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:16.806+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:17.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:16.806+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:17.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:16.806+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:17.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:16.806+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:17.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:16.706+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:17.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:16.706+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:17.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:16.706+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:17.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:16.706+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:18.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:08:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:08:17.794+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 14897 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:08:18.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:17.831+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:18.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:17.831+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:18.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:17.831+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:18.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:17.831+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:18.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:17.824+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:18.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:17.824+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:18.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:17.824+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:18.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:17.824+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:18.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:17.697+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:18.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:17.697+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:18.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:17.697+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:18.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:17.697+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:19.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:18.842+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:19.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:18.842+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:19.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:18.842+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:19.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:18.843+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:19.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:18.787+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:19.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:18.787+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:19.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:18.787+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:19.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:18.787+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:19.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:18.700+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:19.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:18.700+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:19.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:18.700+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:19.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:18.700+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:20.142 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:19.855+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:20.142 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:19.855+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:20.142 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:19.855+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:20.142 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:19.855+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:20.143 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:19.706+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:20.143 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:19.706+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:20.143 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:19.706+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:20.143 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:19.707+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:20.143 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:19.748+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:20.143 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:19.748+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:20.143 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:19.748+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:20.144 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:19.748+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:21.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:20.742+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:21.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:20.742+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:21.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:20.742+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:21.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:20.742+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:21.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:20.754+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:21.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:20.754+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:21.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:20.754+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:21.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:20.754+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:21.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:20.839+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:21.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:20.839+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:21.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:20.839+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:21.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:20.839+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:22.000 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:21.705+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:22.000 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:21.705+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:22.000 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:21.705+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:22.000 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:21.705+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:22.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:21.764+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:22.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:21.764+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:22.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:21.764+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:22.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:21.765+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:22.001 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:21.867+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:22.001 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:21.867+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:22.002 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:21.867+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:22.002 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:21.868+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:23.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:08:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:08:22.796+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 14988 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:08:23.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:22.736+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:23.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:22.736+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:23.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:22.736+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:23.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:22.736+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:23.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:22.735+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:23.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:22.735+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:23.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:22.735+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:23.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:22.735+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:23.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:22.841+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:23.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:22.841+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:23.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:22.841+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:23.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:22.841+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:23.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:23.752+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:23.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:23.753+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:23.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:23.753+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:23.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:23.753+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:23.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:23.773+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:23.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:23.773+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:23.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:23.773+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:23.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:23.774+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:23.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:23.802+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:23.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:23.802+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:23.926 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:23.802+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:23.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:23.802+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:24.464 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-06T03:08:24.464 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2024-04-06T03:08:24.465 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-06T03:08:24.465 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T03:08:24.485 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-06T03:08:24.485 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T03:08:24.545 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-06T03:08:25.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:24.790+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:25.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:24.790+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:25.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:24.790+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:25.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:24.790+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:25.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:24.749+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:25.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:24.749+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:25.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:24.749+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:25.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:24.749+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:25.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:24.852+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:25.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:24.852+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:25.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:24.852+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:25.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:24.852+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:26.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:25.885+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:26.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:25.885+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:26.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:25.885+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:26.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:25.886+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:26.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:25.777+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:26.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:25.777+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:26.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:25.777+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:26.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:25.777+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:26.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:25.711+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:26.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:25.711+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:26.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:25.711+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:26.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:25.711+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:27.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:26.883+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:27.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:26.883+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:27.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:26.883+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:27.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:26.884+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:27.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:26.752+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:27.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:26.752+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:27.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:26.752+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:27.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:26.752+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:27.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:26.742+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:27.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:26.742+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:27.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:26.742+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:27.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:26.742+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:27.778 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:27.722+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:27.779 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:27.722+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:27.779 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:27.722+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:27.779 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:27.722+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:28.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:08:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:08:27.799+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 15079 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:08:28.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:27.793+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:28.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:27.793+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:28.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:27.793+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:28.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:27.793+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:28.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:27.896+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:28.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:27.896+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:28.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:27.896+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:28.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:27.896+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:28.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:28.885+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:28.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:28.885+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:28.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:28.885+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:28.925 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:28.886+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:28.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:28.689+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:28.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:28.689+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:28.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:28.689+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:28.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:28.689+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:28.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:28.831+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:28.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:28.831+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:28.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:28.831+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:28.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:28.831+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:29.682 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:29.695+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:29.682 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:29.695+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:29.682 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:29.695+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:29.682 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:29.695+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:30.142 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:29.879+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:30.143 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:29.879+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:30.143 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:29.879+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:30.143 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:29.879+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:30.143 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:29.932+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:30.143 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:29.933+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:30.144 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:29.933+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:30.144 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:29.933+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:31.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:30.960+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:31.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:30.960+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:31.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:30.960+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:31.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:30.960+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:31.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:30.743+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:31.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:30.743+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:31.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:30.743+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:31.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:30.743+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:31.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:30.834+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:31.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:30.834+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:31.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:30.834+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:31.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:30.834+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:32.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:31.789+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:32.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:31.789+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:32.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:31.789+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:32.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:31.789+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:32.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:31.962+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:32.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:31.962+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:32.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:31.962+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:32.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:31.963+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:32.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:31.857+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:32.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:31.857+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:32.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:31.857+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:32.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:31.857+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:32.801 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:08:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:08:32.801+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 15169 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:08:32.802 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:32.773+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:32.802 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:32.773+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:32.803 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:32.773+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:32.803 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:32.773+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:33.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:32.947+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:33.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:32.947+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:33.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:32.947+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:33.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:32.947+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:33.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:32.886+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:33.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:32.886+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:33.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:32.886+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:33.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:32.886+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:33.848 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:33.766+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:33.848 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:33.766+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:33.848 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:33.766+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:33.848 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:33.766+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:34.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:33.967+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:34.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:33.967+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:34.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:33.967+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:34.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:33.967+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:34.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:33.863+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:34.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:33.863+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:34.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:33.863+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:34.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:33.863+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:35.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:35.005+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:35.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:35.005+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:35.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:35.005+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:35.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:35.005+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:35.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:34.806+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:35.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:34.806+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:35.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:34.806+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:35.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:34.807+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:35.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:34.860+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:35.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:34.860+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:35.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:34.860+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:35.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:34.861+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:36.031 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:35.785+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:36.032 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:35.785+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:36.032 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:35.785+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:36.032 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:35.786+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:36.032 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:35.828+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:36.033 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:35.828+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:36.033 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:35.828+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:36.033 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:35.828+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:36.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:36.046+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:36.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:36.046+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:36.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:36.046+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:36.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:36.046+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:37.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:36.832+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:37.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:36.832+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:37.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:36.832+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:37.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:36.832+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:37.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:36.762+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:37.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:36.762+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:37.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:36.762+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:37.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:36.763+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:37.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:37.006+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:37.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:37.006+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:37.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:37.006+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:37.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:37.006+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:37.915 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:08:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:08:37.804+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 15259 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:08:37.916 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:37.841+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:37.916 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:37.841+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:37.916 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:37.841+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:37.916 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:37.842+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:37.916 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:37.714+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:37.917 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:37.714+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:37.917 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:37.714+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:37.917 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:37.714+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:38.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:38.016+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:38.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:38.016+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:38.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:38.016+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:38.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:38.016+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:38.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:38.667+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:38.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:38.667+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:38.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:38.667+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:38.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:38.667+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:38.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:38.879+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:38.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:38.879+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:38.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:38.879+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:38.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:38.879+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:39.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:39.042+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:39.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:39.042+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:39.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:39.042+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:39.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:39.042+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:39.867 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:39.656+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:39.867 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:39.656+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:39.867 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:39.656+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:39.868 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:39.656+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:40.143 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:39.882+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:40.143 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:39.882+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:40.143 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:39.882+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:40.143 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:39.882+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:40.144 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:40.076+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:40.144 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:40.076+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:40.144 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:40.076+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:40.144 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:40.076+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:40.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:40.685+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:40.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:40.685+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:40.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:40.685+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:40.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:40.685+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:40.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:40.922+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:40.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:40.922+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:40.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:40.922+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:40.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:40.922+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:41.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:41.101+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:41.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:41.101+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:41.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:41.101+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:41.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:41.101+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:42.000 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:41.960+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:42.001 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:41.960+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:42.001 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:41.960+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:42.001 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:41.960+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:42.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:41.719+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:42.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:41.719+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:42.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:41.719+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:42.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:41.719+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:42.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:42.089+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:42.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:42.089+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:42.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:42.089+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:42.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:42.089+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:42.899 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:08:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:08:42.807+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 15349 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:08:42.899 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:42.697+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:42.899 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:42.697+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:42.899 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:42.697+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:42.899 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:42.697+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:43.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:43.074+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:43.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:43.074+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:43.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:43.074+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:43.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:43.074+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:43.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:42.913+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:43.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:42.913+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:43.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:42.913+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:43.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:42.914+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:43.867 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:43.658+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:43.867 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:43.658+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:43.867 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:43.658+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:43.868 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:43.658+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:44.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:44.084+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:44.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:44.084+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:44.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:44.084+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:44.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:44.084+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:44.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:43.882+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:44.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:43.882+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:44.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:43.882+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:44.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:43.882+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:44.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:44.652+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:44.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:44.652+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:44.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:44.652+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:44.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:44.652+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:44.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:44.859+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:44.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:44.859+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:44.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:44.859+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:44.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:44.859+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:45.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:45.083+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:45.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:45.083+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:45.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:45.083+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:45.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:45.083+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:46.042 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:45.698+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:46.043 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:45.698+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:46.043 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:45.698+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:46.043 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:45.698+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:46.043 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:45.824+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:46.043 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:45.824+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:46.043 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:45.824+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:46.043 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:45.824+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:46.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:46.057+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:46.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:46.057+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:46.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:46.057+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:46.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:46.057+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:46.924 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:46.870+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:46.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:46.870+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:46.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:46.870+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:46.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:46.870+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:46.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:46.650+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:46.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:46.650+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:46.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:46.650+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:46.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:46.650+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:47.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:47.020+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:47.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:47.020+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:47.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:47.020+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:47.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:47.020+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:47.920 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:08:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:08:47.809+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 15439 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:08:47.920 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:47.868+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:47.920 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:47.868+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:47.921 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:47.868+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:47.921 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:47.868+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:47.921 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:47.692+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:47.921 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:47.692+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:47.921 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:47.692+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:47.921 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:47.692+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:48.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:48.003+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:48.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:48.003+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:48.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:48.003+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:48.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:48.003+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:49.036 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:48.910+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:49.036 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:48.910+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:49.036 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:48.910+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:49.036 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:48.910+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:49.036 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:48.713+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:49.037 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:48.713+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:49.037 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:48.713+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:49.037 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:48.714+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:49.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:49.050+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:49.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:49.050+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:49.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:49.050+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:49.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:49.050+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:50.025 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:49.760+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:50.025 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:49.760+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:50.025 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:49.760+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:50.026 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:49.760+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:50.026 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:49.945+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:50.026 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:49.945+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:50.026 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:49.945+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:50.026 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:49.945+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:50.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:50.040+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:50.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:50.040+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:50.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:50.040+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:50.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:50.040+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:51.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:50.802+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:51.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:50.802+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:51.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:50.802+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:51.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:50.802+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:51.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:51.030+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:51.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:51.030+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:51.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:51.030+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:51.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:51.030+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:51.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:50.936+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:51.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:50.936+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:51.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:50.936+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:51.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:50.936+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:52.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:51.941+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:52.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:51.941+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:52.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:51.941+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:52.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:51.941+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:52.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:51.775+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:52.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:51.775+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:52.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:51.775+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:52.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:51.776+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:52.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:52.000+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:52.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:52.000+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:52.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:52.000+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:52.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:52.000+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:52.985 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:08:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:08:52.811+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 15528 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:08:52.986 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:52.746+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:52.986 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:52.746+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:52.986 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:52.746+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:52.986 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:52.747+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:52.987 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:52.913+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:52.987 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:52.913+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:52.987 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:52.913+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:52.987 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:52.913+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:53.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:53.000+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:53.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:53.000+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:53.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:53.000+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:53.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:53.000+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:53.849 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:53.720+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:53.849 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:53.720+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:53.849 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:53.720+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:53.850 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:53.720+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:54.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:54.024+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:54.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:54.024+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:54.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:54.024+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:54.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:54.024+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:54.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:53.864+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:54.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:53.864+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:54.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:53.864+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:54.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:53.864+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:54.848 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:54.681+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:54.848 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:54.681+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:54.848 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:54.681+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:54.848 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:54.681+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:55.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:54.863+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:55.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:54.863+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:55.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:54.863+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:55.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:54.863+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:55.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:54.995+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:55.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:54.995+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:55.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:54.995+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:55.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:54.995+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:56.008 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:55.729+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:56.009 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:55.729+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:56.009 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:55.729+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:56.009 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:55.729+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:56.009 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:55.832+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:56.010 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:55.832+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:56.010 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:55.832+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:56.010 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:55.832+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:56.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:56.024+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:56.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:56.024+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:56.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:56.024+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:56.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:56.024+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:57.004 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:56.824+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:57.004 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:56.824+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:57.004 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:56.824+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:57.005 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:56.824+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:57.005 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:56.749+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:57.005 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:56.749+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:57.005 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:56.749+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:57.005 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:56.749+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:57.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:57.019+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:57.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:57.019+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:57.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:57.019+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:57.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:57.019+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:58.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:08:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:08:57.814+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 15618 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:08:58.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:57.837+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:58.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:57.837+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:58.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:57.837+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:58.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:57.837+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:58.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:57.755+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:58.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:57.755+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:58.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:57.755+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:58.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:57.755+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:58.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:57.985+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:58.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:57.985+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:58.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:57.985+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:58.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:57.985+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:08:59.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:58.801+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:59.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:58.801+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:59.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:58.801+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:08:59.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:58.802+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:59.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:58.796+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:59.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:58.796+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:59.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:58.796+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:08:59.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:58.796+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:08:59.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:59.020+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:59.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:59.020+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:59.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:59.020+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:08:59.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:08:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:08:59.020+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:00.037 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:59.789+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:00.037 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:59.789+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:00.038 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:59.789+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:00.038 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:08:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:08:59.789+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:00.038 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:59.836+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:00.038 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:59.836+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:00.038 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:59.836+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:00.038 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:08:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:08:59.837+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:00.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:00.052+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:00.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:00.052+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:00.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:00.052+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:00.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:00.052+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:00.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:00.806+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:00.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:00.806+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:00.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:00.806+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:00.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:00.806+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:00.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:00.865+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:00.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:00.865+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:00.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:00.865+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:00.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:00.865+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:01.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:01.065+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:01.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:01.065+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:01.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:01.065+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:01.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:01.065+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:02.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:01.852+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:02.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:01.852+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:02.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:01.852+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:02.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:01.852+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:02.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:02.058+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:02.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:02.058+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:02.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:02.058+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:02.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:02.058+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:02.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:01.894+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:02.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:01.894+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:02.178 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:01.894+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:02.178 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:01.894+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:03.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:09:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:09:02.815+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 15709 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:09:03.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:03.010+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:03.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:03.010+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:03.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:03.010+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:03.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:03.010+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:03.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:02.808+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:03.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:02.808+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:03.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:02.808+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:03.177 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:02.808+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:03.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:02.850+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:03.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:02.850+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:03.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:02.850+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:03.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:02.850+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:03.874 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:03.818+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:03.875 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:03.818+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:03.875 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:03.818+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:03.875 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:03.819+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:04.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:04.011+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:04.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:04.011+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:04.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:04.011+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:04.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:04.012+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:04.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:03.889+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:04.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:03.889+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:04.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:03.889+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:04.177 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:03.889+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:05.174 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:04.909+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:05.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:04.909+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:05.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:04.909+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:05.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:04.910+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:05.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:04.819+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:05.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:04.819+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:05.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:04.819+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:05.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:04.819+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:05.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:05.053+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:05.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:05.053+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:05.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:05.053+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:05.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:05.054+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:06.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:05.890+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:06.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:05.890+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:06.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:05.890+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:06.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:05.890+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:06.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:05.808+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:06.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:05.808+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:06.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:05.808+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:06.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:05.808+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:06.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:06.039+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:06.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:06.039+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:06.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:06.039+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:06.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:06.039+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:07.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:06.916+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:07.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:06.916+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:07.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:06.916+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:07.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:06.916+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:07.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:06.841+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:07.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:06.841+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:07.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:06.841+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:07.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:06.841+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:07.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:07.063+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:07.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:07.063+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:07.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:07.063+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:07.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:07.063+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:08.044 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:09:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:09:07.818+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 15799 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:09:08.045 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:07.809+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:08.045 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:07.809+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:08.045 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:07.809+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:08.045 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:07.809+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:08.046 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:07.907+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:08.046 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:07.907+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:08.046 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:07.907+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:08.046 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:07.907+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:08.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:08.059+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:08.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:08.059+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:08.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:08.059+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:08.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:08.059+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:09.066 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:08.894+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:09.066 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:08.894+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:09.066 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:08.894+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:09.066 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:08.894+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:09.066 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:08.819+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:09.067 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:08.819+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:09.067 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:08.819+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:09.067 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:08.819+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:09.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:09.080+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:09.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:09.080+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:09.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:09.080+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:09.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:09.080+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:10.104 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:09.863+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:10.104 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:09.863+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:10.105 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:09.863+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:10.105 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:09.863+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:10.105 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:09.827+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:10.105 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:09.827+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:10.105 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:09.827+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:10.105 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:09.828+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:10.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:10.119+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:10.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:10.119+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:10.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:10.119+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:10.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:10.119+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:11.135 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:10.881+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:11.135 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:10.881+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:11.135 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:10.881+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:11.135 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:10.881+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:11.136 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:10.843+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:11.136 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:10.843+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:11.136 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:10.843+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:11.136 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:10.843+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:11.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:11.149+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:11.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:11.149+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:11.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:11.149+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:11.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:11.150+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:12.091 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:11.868+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:12.091 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:11.868+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:12.091 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:11.868+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:12.091 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:11.868+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:12.092 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:11.805+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:12.092 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:11.805+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:12.092 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:11.805+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:12.092 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:11.806+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:12.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:12.105+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:12.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:12.105+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:12.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:12.105+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:12.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:12.106+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:13.084 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:09:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:09:12.821+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 15887 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:09:13.085 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:12.892+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:13.085 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:12.892+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:13.085 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:12.892+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:13.085 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:12.893+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:13.085 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:12.804+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:13.086 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:12.804+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:13.086 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:12.804+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:13.086 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:12.804+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:13.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:13.099+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:13.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:13.099+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:13.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:13.099+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:13.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:13.099+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:13.885 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:13.819+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:13.885 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:13.819+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:13.885 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:13.819+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:13.885 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:13.820+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:14.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:14.071+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:14.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:14.071+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:14.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:14.071+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:14.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:14.071+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:14.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:13.899+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:14.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:13.899+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:14.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:13.899+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:14.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:13.899+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:15.084 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:14.832+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:15.084 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:14.832+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:15.084 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:14.832+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:15.084 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:14.832+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:15.084 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:14.936+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:15.085 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:14.936+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:15.085 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:14.936+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:15.085 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:14.936+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:15.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:15.098+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:15.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:15.098+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:15.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:15.098+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:15.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:15.099+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:16.080 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:15.828+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:16.081 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:15.828+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:16.081 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:15.828+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:16.081 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:15.829+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:16.081 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:15.942+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:16.081 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:15.942+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:16.081 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:15.942+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:16.082 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:15.942+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:16.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:16.095+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:16.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:16.095+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:16.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:16.095+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:16.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:16.095+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:17.127 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:16.868+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:17.127 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:16.868+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:17.127 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:16.868+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:17.128 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:16.868+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:17.128 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:16.904+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:17.128 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:16.904+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:17.128 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:16.904+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:17.128 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:16.904+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:17.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:17.142+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:17.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:17.142+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:17.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:17.142+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:17.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:17.142+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:18.093 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:09:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:09:17.824+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 15978 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:09:18.093 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:17.857+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:18.093 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:17.857+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:18.094 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:17.857+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:18.094 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:17.858+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:18.094 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:17.842+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:18.094 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:17.842+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:18.094 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:17.842+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:18.094 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:17.842+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:18.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:18.107+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:18.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:18.107+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:18.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:18.107+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:18.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:18.108+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:19.078 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:18.832+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:19.078 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:18.832+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:19.079 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:18.832+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:19.079 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:18.832+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:19.079 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:18.859+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:19.079 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:18.859+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:19.079 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:18.859+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:19.079 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:18.859+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:19.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:19.124+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:19.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:19.124+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:19.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:19.124+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:19.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:19.125+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:20.097 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:19.793+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:20.097 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:19.793+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:20.098 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:19.793+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:20.098 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:19.793+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:20.098 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:19.870+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:20.098 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:19.870+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:20.098 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:19.870+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:20.098 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:19.870+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:20.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:20.112+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:20.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:20.112+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:20.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:20.112+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:20.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:20.112+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:21.102 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:20.839+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:21.103 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:20.839+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:21.103 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:20.839+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:21.103 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:20.839+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:21.103 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:20.836+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:21.103 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:20.836+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:21.103 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:20.836+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:21.104 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:20.836+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:21.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:21.117+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:21.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:21.117+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:21.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:21.117+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:21.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:21.117+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:22.149 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:21.889+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:22.150 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:21.889+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:22.150 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:21.889+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:22.150 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:21.889+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:22.150 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:22.103+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:22.151 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:22.103+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:22.151 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:22.103+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:22.151 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:22.103+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:22.151 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:21.875+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:22.151 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:21.875+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:22.151 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:21.875+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:22.151 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:21.875+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:23.118 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:09:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:09:22.826+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 16067 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:09:23.118 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:22.911+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:23.119 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:22.911+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:23.119 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:22.911+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:23.119 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:22.911+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:23.119 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:22.869+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:23.119 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:22.869+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:23.119 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:22.869+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:23.120 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:22.869+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:23.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:23.132+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:23.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:23.132+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:23.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:23.132+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:23.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:23.133+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:24.133 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:23.874+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:24.134 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:23.874+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:24.134 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:23.874+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:24.134 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:23.875+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:24.134 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:23.851+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:24.134 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:23.851+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:24.134 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:23.851+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:24.135 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:23.851+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:24.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:24.148+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:24.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:24.148+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:24.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:24.148+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:24.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:24.148+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:25.130 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:24.870+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:25.130 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:24.870+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:25.130 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:24.870+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:25.130 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:24.870+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:25.131 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:24.881+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:25.131 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:24.881+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:25.131 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:24.881+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:25.131 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:24.882+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:25.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:25.144+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:25.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:25.144+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:25.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:25.144+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:25.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:25.145+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:25.905 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:25.831+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:25.906 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:25.831+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:25.906 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:25.831+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:25.906 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:25.831+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:26.169 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:25.920+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:26.169 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:25.920+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:26.169 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:25.920+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:26.169 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:25.920+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:26.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:26.183+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:26.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:26.183+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:26.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:26.183+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:26.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:26.184+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:27.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:26.871+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:27.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:26.871+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:27.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:26.871+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:27.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:26.871+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:27.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:26.794+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:27.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:26.794+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:27.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:26.794+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:27.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:26.795+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:27.674 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:27.214+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:27.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:27.214+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:27.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:27.214+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:27.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:27.214+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:28.007 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:09:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:09:27.829+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 16158 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:09:28.008 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:27.821+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:28.008 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:27.821+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:28.008 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:27.821+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:28.008 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:27.822+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:28.009 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:27.767+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:28.009 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:27.767+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:28.009 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:27.767+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:28.009 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:27.767+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:28.294 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:28.209+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:28.294 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:28.209+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:28.294 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:28.209+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:28.294 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:28.209+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:29.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:28.836+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:29.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:28.836+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:29.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:28.836+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:29.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:28.836+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:29.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:28.737+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:29.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:28.737+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:29.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:28.737+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:29.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:28.737+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:29.674 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:29.222+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:29.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:29.222+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:29.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:29.222+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:29.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:29.222+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:30.147 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:29.799+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:30.147 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:29.799+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:30.147 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:29.799+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:30.147 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:29.800+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:30.148 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:29.700+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:30.148 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:29.700+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:30.148 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:29.700+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:30.148 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:29.700+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:30.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:30.235+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:30.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:30.235+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:30.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:30.235+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:30.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:30.236+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:31.174 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:30.836+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:31.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:30.836+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:31.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:30.836+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:31.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:30.836+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:31.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:30.728+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:31.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:30.728+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:31.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:30.728+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:31.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:30.728+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:31.674 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:31.276+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:31.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:31.276+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:31.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:31.276+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:31.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:31.277+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:32.174 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:31.798+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:32.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:31.798+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:32.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:31.798+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:32.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:31.798+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:32.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:31.771+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:32.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:31.771+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:32.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:31.771+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:32.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:31.771+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:32.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:32.324+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:32.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:32.324+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:32.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:32.324+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:32.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:32.324+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:33.049 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:09:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:09:32.832+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 16246 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:09:33.049 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:32.809+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:33.050 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:32.809+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:33.050 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:32.809+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:33.050 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:32.809+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:33.050 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:32.733+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:33.050 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:32.733+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:33.050 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:32.733+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:33.051 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:32.733+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:33.672 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:33.362+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:33.672 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:33.362+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:33.672 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:33.362+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:33.672 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:33.362+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:33.924 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:33.777+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:33.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:33.777+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:33.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:33.777+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:33.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:33.777+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:33.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:33.686+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:33.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:33.686+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:33.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:33.686+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:33.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:33.687+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:34.674 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:34.317+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:34.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:34.317+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:34.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:34.317+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:34.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:34.317+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:35.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:34.791+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:35.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:34.791+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:35.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:34.791+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:35.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:34.791+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:35.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:34.733+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:35.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:34.733+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:35.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:34.733+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:35.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:34.733+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:35.674 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:35.299+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:35.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:35.299+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:35.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:35.299+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:35.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:35.299+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:36.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:35.811+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:36.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:35.811+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:36.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:35.811+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:36.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:35.812+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:36.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:35.742+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:36.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:35.742+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:36.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:35.742+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:36.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:35.742+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:36.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:36.257+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:36.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:36.257+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:36.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:36.257+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:36.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:36.257+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:37.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:36.835+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:37.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:36.835+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:37.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:36.835+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:37.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:36.835+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:37.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:36.717+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:37.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:36.717+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:37.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:36.717+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:37.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:36.717+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:37.510 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:37.254+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:37.510 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:37.254+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:37.510 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:37.254+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:37.511 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:37.254+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:37.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:09:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:09:37.834+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 16338 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:09:37.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:37.824+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:37.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:37.824+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:37.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:37.824+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:37.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:37.824+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:37.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:37.684+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:37.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:37.684+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:37.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:37.684+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:37.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:37.684+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:38.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:38.231+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:38.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:38.231+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:38.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:38.231+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:38.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:38.231+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:39.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:38.850+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:39.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:38.850+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:39.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:38.850+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:39.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:38.850+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:39.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:38.725+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:39.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:38.725+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:39.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:38.725+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:39.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:38.725+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:39.674 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:39.259+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:39.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:39.259+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:39.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:39.259+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:39.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:39.259+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:40.147 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:39.843+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:40.148 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:39.843+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:40.148 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:39.843+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:40.148 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:39.843+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:40.148 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:39.736+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:40.149 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:39.736+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:40.149 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:39.736+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:40.149 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:39.736+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:40.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:40.246+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:40.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:40.246+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:40.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:40.246+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:40.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:40.246+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:40.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:40.803+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:40.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:40.803+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:40.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:40.803+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:40.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:40.803+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:40.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:40.702+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:40.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:40.702+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:40.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:40.702+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:40.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:40.702+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:41.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:41.287+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:41.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:41.287+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:41.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:41.287+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:41.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:41.288+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:42.000 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:41.852+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:42.000 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:41.852+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:42.000 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:41.852+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:42.000 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:41.852+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:42.001 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:41.732+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:42.001 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:41.732+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:42.001 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:41.732+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:42.001 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:41.732+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:42.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:42.337+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:42.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:42.337+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:42.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:42.337+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:42.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:42.337+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:43.133 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:09:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:09:42.837+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 16428 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:09:43.134 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:42.860+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:43.134 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:42.860+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:43.134 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:42.860+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:43.134 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:42.860+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:43.135 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:42.742+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:43.135 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:42.742+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:43.135 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:42.742+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:43.135 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:42.743+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:43.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:43.295+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:43.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:43.295+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:43.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:43.295+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:43.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:43.295+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:43.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:43.724+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:43.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:43.724+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:43.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:43.724+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:43.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:43.724+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:43.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:43.812+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:43.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:43.812+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:43.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:43.812+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:43.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:43.812+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:44.661 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:44.290+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:44.661 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:44.290+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:44.661 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:44.290+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:44.661 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:44.291+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:44.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:44.801+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:44.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:44.801+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:44.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:44.801+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:44.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:44.801+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:44.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:44.676+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:44.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:44.676+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:44.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:44.676+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:44.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:44.676+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:45.674 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:45.253+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:45.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:45.253+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:45.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:45.253+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:45.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:45.253+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:46.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:45.799+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:46.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:45.799+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:46.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:45.799+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:46.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:45.799+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:46.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:45.718+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:46.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:45.718+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:46.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:45.718+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:46.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:45.718+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:46.674 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:46.248+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:46.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:46.248+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:46.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:46.248+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:46.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:46.248+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:47.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:46.849+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:47.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:46.849+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:47.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:46.849+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:47.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:46.849+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:47.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:46.727+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:47.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:46.727+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:47.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:46.727+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:47.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:46.727+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:47.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:47.289+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:47.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:47.289+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:47.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:47.289+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:47.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:47.289+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:48.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:09:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:09:47.839+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 16519 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:09:48.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:47.861+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:48.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:47.861+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:48.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:47.861+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:48.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:47.861+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:48.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:47.742+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:48.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:47.742+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:48.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:47.742+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:48.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:47.742+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:48.674 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:48.290+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:48.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:48.290+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:48.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:48.290+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:48.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:48.290+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:49.174 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:48.864+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:49.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:48.864+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:49.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:48.864+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:49.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:48.864+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:49.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:48.731+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:49.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:48.731+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:49.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:48.731+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:49.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:48.731+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:49.674 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:49.293+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:49.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:49.293+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:49.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:49.293+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:49.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:49.294+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:50.147 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:49.741+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:50.147 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:49.741+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:50.148 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:49.741+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:50.148 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:49.741+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:50.148 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:49.825+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:50.148 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:49.825+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:50.148 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:49.825+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:50.148 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:49.825+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:50.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:50.328+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:50.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:50.328+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:50.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:50.328+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:50.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:50.328+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:51.174 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:50.776+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:51.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:50.776+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:51.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:50.776+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:51.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:50.776+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:51.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:50.771+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:51.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:50.771+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:51.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:50.771+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:51.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:50.772+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:51.674 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:51.294+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:51.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:51.294+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:51.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:51.294+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:51.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:51.294+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:52.174 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:51.810+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:52.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:51.810+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:52.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:51.810+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:52.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:51.810+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:52.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:51.801+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:52.175 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:51.801+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:52.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:51.801+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:52.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:51.801+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:52.674 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:52.265+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:52.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:52.265+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:52.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:52.265+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:52.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:52.265+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:53.174 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:09:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:09:52.842+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 16606 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:09:53.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:52.781+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:53.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:52.781+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:53.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:52.781+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:53.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:52.781+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:53.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:52.815+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:53.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:52.816+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:53.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:52.816+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:53.176 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:52.816+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:53.584 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:53.218+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:53.584 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:53.218+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:53.584 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:53.218+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:53.584 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:53.219+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:53.924 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:53.733+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:53.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:53.734+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:53.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:53.734+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:53.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:53.734+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:53.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:53.820+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:53.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:53.820+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:53.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:53.820+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:53.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:53.820+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:54.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:54.188+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:54.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:54.188+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:54.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:54.188+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:54.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:54.188+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:55.143 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:54.764+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:55.143 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:54.764+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:55.143 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:54.764+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:55.143 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:54.764+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:55.143 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:54.792+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:55.144 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:54.792+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:55.144 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:54.792+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:55.144 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:54.792+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:55.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:55.157+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:55.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:55.157+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:55.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:55.157+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:55.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:55.158+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:56.157 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:55.779+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:56.158 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:55.779+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:56.158 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:55.779+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:56.158 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:55.779+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:56.159 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:55.779+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:56.159 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:55.779+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:56.159 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:55.779+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:56.159 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:55.779+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:56.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:56.172+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:56.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:56.172+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:56.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:56.172+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:56.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:56.173+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:57.141 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:56.749+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:57.141 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:56.749+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:57.142 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:56.749+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:57.142 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:56.749+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:57.142 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:56.739+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:57.142 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:56.739+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:57.142 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:56.739+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:57.143 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:56.739+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:57.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:57.156+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:57.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:57.156+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:57.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:57.156+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:57.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:57.156+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:57.956 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:09:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:09:57.845+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 16696 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:09:57.957 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:57.732+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:57.957 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:57.732+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:57.957 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:57.732+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:57.957 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:57.733+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:57.958 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:57.697+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:57.958 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:57.697+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:57.958 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:57.697+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:57.958 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:57.698+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:58.259 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:58.182+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:58.259 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:58.182+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:58.259 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:58.182+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:58.259 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:58.182+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:58.924 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:58.676+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:58.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:58.676+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:58.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:58.676+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:58.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:58.676+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:58.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:58.693+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:58.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:58.693+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:58.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:58.693+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:58.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:58.694+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:59.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:59.136+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:59.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:59.136+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:59.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:59.136+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:09:59.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:09:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:09:59.137+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:09:59.924 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:59.683+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:59.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:59.683+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:59.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:59.683+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:09:59.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:09:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:09:59.684+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:09:59.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:59.666+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:59.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:59.666+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:59.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:59.666+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:09:59.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:09:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:09:59.666+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:00.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:00.108+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:00.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:00.108+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:00.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:00.108+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:00.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:00.108+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:00.924 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:00.637+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:00.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:00.637+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:00.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:00.637+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:00.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:00.638+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:00.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:00.666+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:00.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:00.666+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:00.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:00.666+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:00.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:00.666+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:01.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:01.071+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:01.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:01.072+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:01.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:01.072+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:01.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:01.072+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:01.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:01.624+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:01.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:01.624+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:01.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:01.624+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:01.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:01.624+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:01.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:01.678+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:01.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:01.678+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:01.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:01.678+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:01.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:01.679+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:02.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:02.041+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:02.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:02.041+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:02.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:02.041+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:02.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:02.041+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:02.924 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:10:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:10:02.848+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 16784 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:10:02.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:02.669+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:02.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:02.669+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:02.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:02.669+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:02.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:02.670+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:02.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:02.633+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:02.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:02.633+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:02.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:02.633+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:02.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:02.634+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:03.301 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:03.005+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:03.301 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:03.005+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:03.301 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:03.005+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:03.301 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:03.005+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:03.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:03.650+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:03.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:03.650+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:03.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:03.650+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:03.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:03.650+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:03.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:03.629+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:03.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:03.629+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:03.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:03.629+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:03.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:03.629+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:04.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:04.038+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:04.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:04.038+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:04.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:04.038+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:04.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:04.038+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:04.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:04.615+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:04.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:04.615+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:04.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:04.615+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:04.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:04.615+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:04.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:04.608+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:04.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:04.608+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:04.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:04.608+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:04.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:04.608+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:05.424 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:05.087+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:05.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:05.087+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:05.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:05.087+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:05.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:05.087+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:05.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:05.638+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:05.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:05.638+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:05.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:05.638+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:05.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:05.638+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:05.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:05.577+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:05.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:05.577+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:05.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:05.577+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:05.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:05.577+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:06.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:06.092+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:06.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:06.092+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:06.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:06.092+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:06.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:06.092+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:06.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:06.629+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:06.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:06.629+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:06.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:06.629+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:06.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:06.629+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:06.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:06.612+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:06.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:06.612+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:06.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:06.612+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:06.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:06.612+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:07.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:07.106+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:07.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:07.106+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:07.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:07.106+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:07.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:07.107+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:07.925 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:10:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:10:07.850+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 16874 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:10:07.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:07.653+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:07.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:07.653+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:07.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:07.653+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:07.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:07.653+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:07.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:07.626+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:07.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:07.626+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:07.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:07.626+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:07.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:07.626+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:08.345 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:08.096+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:08.345 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:08.096+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:08.345 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:08.096+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:08.345 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:08.096+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:08.635 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:08.597+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:08.635 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:08.597+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:08.635 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:08.597+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:08.635 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:08.598+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:08.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:08.649+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:08.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:08.649+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:08.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:08.649+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:08.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:08.649+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:09.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:09.080+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:09.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:09.080+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:09.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:09.080+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:09.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:09.080+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:09.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:09.639+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:09.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:09.639+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:09.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:09.639+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:09.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:09.639+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:09.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:09.608+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:09.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:09.608+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:09.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:09.608+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:09.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:09.609+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:10.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:10.115+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:10.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:10.115+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:10.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:10.115+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:10.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:10.115+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:10.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:10.606+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:10.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:10.606+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:10.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:10.606+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:10.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:10.606+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:10.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:10.564+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:10.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:10.564+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:10.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:10.565+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:10.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:10.565+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:11.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:11.124+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:11.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:11.124+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:11.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:11.124+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:11.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:11.124+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:11.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:11.574+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:11.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:11.574+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:11.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:11.574+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:11.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:11.574+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:11.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:11.600+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:11.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:11.600+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:11.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:11.600+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:11.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:11.600+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:12.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:12.169+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:12.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:12.169+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:12.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:12.169+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:12.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:12.169+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:12.839 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:12.550+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:12.839 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:12.550+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:12.840 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:12.550+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:12.840 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:12.550+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:12.840 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:12.638+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:12.840 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:12.638+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:12.840 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:12.638+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:12.840 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:12.639+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:13.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:10:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:10:12.853+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 16961 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:10:13.565 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:13.197+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:13.565 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:13.197+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:13.565 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:13.197+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:13.566 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:13.198+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:13.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:13.625+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:13.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:13.625+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:13.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:13.625+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:13.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:13.626+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:13.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:13.579+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:13.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:13.579+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:13.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:13.579+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:13.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:13.579+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:14.612 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:14.248+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:14.612 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:14.248+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:14.612 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:14.248+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:14.612 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:14.248+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:14.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:14.667+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:14.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:14.667+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:14.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:14.667+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:14.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:14.667+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:14.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:14.626+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:14.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:14.626+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:14.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:14.626+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:14.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:14.626+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:15.626 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:15.289+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:15.626 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:15.289+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:15.627 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:15.289+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:15.627 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:15.289+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:15.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:15.712+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:15.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:15.712+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:15.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:15.712+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:15.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:15.712+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:15.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:15.641+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:15.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:15.641+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:15.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:15.641+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:15.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:15.641+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:16.615 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:16.330+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:16.616 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:16.330+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:16.616 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:16.330+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:16.616 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:16.330+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:16.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:16.720+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:16.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:16.720+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:16.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:16.720+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:16.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:16.720+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:16.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:16.630+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:16.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:16.630+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:16.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:16.630+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:16.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:16.630+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:17.593 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:17.334+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:17.593 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:17.334+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:17.594 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:17.334+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:17.594 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:17.334+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:17.855 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:10:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:10:17.856+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 17051 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:10:17.855 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:17.745+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:17.855 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:17.745+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:17.856 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:17.745+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:17.856 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:17.746+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:17.856 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:17.607+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:17.856 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:17.607+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:17.856 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:17.607+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:17.856 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:17.608+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:18.563 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:18.301+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:18.564 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:18.301+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:18.564 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:18.301+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:18.564 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:18.301+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:18.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:18.717+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:18.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:18.717+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:18.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:18.717+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:18.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:18.717+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:18.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:18.578+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:18.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:18.578+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:18.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:18.578+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:18.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:18.578+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:19.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:19.334+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:19.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:19.334+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:19.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:19.334+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:19.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:19.334+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:19.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:19.529+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:19.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:19.529+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:19.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:19.529+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:19.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:19.529+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:20.150 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:19.722+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:20.150 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:19.722+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:20.151 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:19.722+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:20.151 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:19.723+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:20.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:20.307+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:20.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:20.307+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:20.425 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:20.307+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:20.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:20.307+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:20.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:20.744+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:20.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:20.744+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:20.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:20.744+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:20.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:20.744+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:20.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:20.513+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:20.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:20.513+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:20.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:20.513+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:20.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:20.513+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:21.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:21.357+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:21.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:21.357+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:21.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:21.357+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:21.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:21.358+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:21.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:21.513+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:21.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:21.513+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:21.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:21.513+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:21.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:21.513+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:22.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:21.748+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:22.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:21.748+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:22.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:21.748+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:22.002 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:21.749+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:22.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:22.332+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:22.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:22.332+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:22.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:22.332+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:22.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:22.332+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:22.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:22.496+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:22.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:22.496+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:22.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:22.496+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:22.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:22.497+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:22.973 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:10:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:10:22.859+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 17140 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:10:22.973 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:22.705+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:22.973 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:22.705+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:22.973 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:22.705+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:22.974 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:22.705+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:23.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:23.381+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:23.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:23.381+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:23.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:23.381+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:23.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:23.382+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:23.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:23.533+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:23.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:23.533+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:23.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:23.533+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:23.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:23.534+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:24.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:23.693+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:24.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:23.693+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:24.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:23.694+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:24.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:23.694+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:24.623 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-06T03:10:24.624 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2024-04-06T03:10:24.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T03:10:24.624 DEBUG:teuthology.orchestra.run.smithi105:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2024-04-06T03:10:24.654 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:24.403+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:24.654 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:24.403+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:24.654 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:24.403+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:24.654 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:24.403+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:24.654 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:24.491+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:24.655 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:24.491+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:24.655 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:24.491+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:24.655 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:24.491+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:24.660 INFO:tasks.workunit.client.0.smithi105.stderr:/usr/bin/ld: final link failed: Stale file handle 2024-04-06T03:10:24.660 INFO:tasks.workunit.client.0.smithi105.stderr:collect2: error: ld returned 1 exit status 2024-04-06T03:10:24.661 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-06T03:10:24.662 INFO:tasks.workunit.client.0.smithi105.stderr:make: *** [: fsstress] Error 1 2024-04-06T03:10:24.662 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-04-06T03:10:24.662 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-06T03:10:24.676 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi105.front.sepia.ceph.com 2024-04-06T03:10:24.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-06T03:10:24.677 DEBUG:teuthology.orchestra.run.smithi105:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-06T03:10:24.725 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-04-06T03:10:24.725 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-06T03:10:24.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:24.714+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:24.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:24.714+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:24.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:24.714+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:24.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:24.714+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:25.673 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:25.497+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:25.674 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:25.497+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:25.674 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:25.497+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:25.674 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:25.497+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:25.674 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:25.401+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:25.674 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:25.401+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:25.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:25.401+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:25.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:25.401+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:25.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:25.687+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:25.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:25.687+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:25.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:25.687+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:25.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:25.688+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:26.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:26.426+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:26.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:26.426+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:26.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:26.426+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:26.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:26.426+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:26.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:26.473+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:26.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:26.473+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:26.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:26.473+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:26.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:26.473+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:26.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:26.662+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:26.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:26.662+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:26.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:26.662+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:26.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:26.662+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:27.848 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:27.466+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:27.848 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:27.466+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:27.848 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:27.466+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:27.848 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:27.466+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:27.849 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:27.467+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:27.849 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:27.467+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:27.849 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:27.467+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:27.849 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:27.467+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:27.849 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:27.629+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:27.849 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:27.629+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:27.850 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:27.629+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:27.850 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:27.629+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:28.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:10:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:10:27.861+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 17230 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:10:28.675 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:28.436+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:28.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:28.436+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:28.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:28.436+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:28.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:28.436+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:28.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:28.659+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:28.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:28.659+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:28.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:28.659+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:28.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:28.659+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:28.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:28.493+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:28.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:28.493+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:28.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:28.493+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:28.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:28.493+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:29.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:29.654+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:29.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:29.654+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:29.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:29.654+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:29.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:29.654+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:29.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:29.426+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:29.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:29.426+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:29.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:29.426+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:29.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:29.426+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:29.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:29.473+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:29.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:29.473+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:29.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:29.473+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:29.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:29.473+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:30.414 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:30.396+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:30.415 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:30.396+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:30.415 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:30.396+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:30.415 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:30.397+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:30.675 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:30.638+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:30.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:30.638+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:30.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:30.638+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:30.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:30.638+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:30.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:30.428+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:30.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:30.428+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:30.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:30.428+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:30.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:30.429+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:31.644 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:31.407+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:31.644 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:31.407+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:31.644 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:31.407+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:31.644 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:31.407+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:31.645 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:31.440+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:31.645 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:31.440+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:31.645 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:31.440+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:31.645 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:31.440+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:31.925 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:31.658+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:31.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:31.658+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:31.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:31.658+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:31.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:31.658+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:32.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:32.447+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:32.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:32.447+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:32.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:32.447+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:32.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:32.447+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:32.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:32.674+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:32.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:32.674+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:32.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:32.674+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:32.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:32.674+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:32.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:32.434+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:32.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:32.434+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:32.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:32.434+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:32.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:32.434+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:33.175 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:10:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:10:32.864+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 17317 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:10:33.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:33.403+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:33.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:33.403+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:33.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:33.403+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:33.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:33.404+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:33.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:33.626+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:33.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:33.626+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:33.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:33.626+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:33.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:33.626+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:33.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:33.477+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:33.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:33.477+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:33.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:33.477+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:33.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:33.477+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:34.675 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:34.400+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:34.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:34.400+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:34.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:34.400+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:34.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:34.400+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:34.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:34.590+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:34.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:34.590+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:34.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:34.590+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:34.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:34.590+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:34.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:34.436+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:34.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:34.436+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:34.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:34.436+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:34.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:34.437+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:35.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:35.446+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:35.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:35.446+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:35.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:35.446+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:35.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:35.446+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:35.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:35.635+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:35.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:35.635+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:35.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:35.635+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:35.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:35.635+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:35.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:35.411+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:35.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:35.411+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:35.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:35.411+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:35.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:35.412+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:36.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:36.647+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:36.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:36.647+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:36.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:36.647+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:36.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:36.647+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:36.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:36.492+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:36.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:36.492+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:36.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:36.492+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:36.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:36.493+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:36.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:36.415+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:36.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:36.415+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:36.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:36.415+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:36.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:36.415+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:37.613 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:37.498+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:37.614 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:37.498+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:37.614 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:37.498+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:37.614 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:37.499+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:37.614 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:37.373+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:37.614 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:37.373+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:37.614 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:37.373+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:37.615 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:37.373+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:37.875 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:10:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:10:37.867+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 17408 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:10:37.876 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:37.628+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:37.876 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:37.628+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:37.876 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:37.628+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:37.876 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:37.628+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:38.622 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:38.451+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:38.623 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:38.451+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:38.623 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:38.451+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:38.623 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:38.451+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:38.623 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:38.375+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:38.623 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:38.375+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:38.624 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:38.375+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:38.624 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:38.375+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:38.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:38.636+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:38.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:38.636+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:38.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:38.636+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:38.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:38.636+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:39.629 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:39.439+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:39.630 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:39.439+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:39.630 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:39.439+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:39.630 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:39.439+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:39.630 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:39.344+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:39.631 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:39.344+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:39.631 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:39.344+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:39.631 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:39.344+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:39.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:39.642+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:39.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:39.642+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:39.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:39.642+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:39.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:39.642+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:40.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:40.408+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:40.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:40.408+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:40.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:40.408+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:40.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:40.408+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:40.426 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:40.339+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:40.427 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:40.339+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:40.427 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:40.339+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:40.427 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:40.339+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:40.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:40.626+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:40.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:40.626+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:40.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:40.626+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:40.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:40.626+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:41.599 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:41.397+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:41.600 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:41.397+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:41.600 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:41.397+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:41.600 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:41.398+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:41.600 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:41.351+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:41.600 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:41.351+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:41.601 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:41.351+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:41.601 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:41.351+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:41.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:41.613+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:41.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:41.613+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:41.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:41.613+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:41.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:41.613+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:42.636 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:42.369+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:42.636 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:42.369+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:42.636 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:42.369+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:42.637 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:42.370+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:42.637 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:42.347+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:42.637 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:42.347+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:42.637 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:42.347+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:42.637 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:42.348+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:42.926 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:10:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:10:42.870+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 17497 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:10:42.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:42.649+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:42.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:42.649+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:42.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:42.649+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:42.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:42.650+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:43.618 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:43.412+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:43.619 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:43.412+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:43.619 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:43.412+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:43.619 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:43.413+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:43.619 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:43.359+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:43.619 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:43.359+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:43.619 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:43.359+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:43.620 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:43.359+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:43.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:43.632+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:43.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:43.632+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:43.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:43.632+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:43.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:43.632+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:44.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:44.598+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:44.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:44.598+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:44.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:44.598+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:44.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:44.598+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:44.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:44.419+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:44.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:44.419+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:44.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:44.419+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:44.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:44.419+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:44.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:44.381+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:44.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:44.381+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:44.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:44.381+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:44.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:44.381+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:45.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:45.405+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:45.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:45.405+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:45.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:45.405+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:45.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:45.405+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:45.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:45.623+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:45.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:45.623+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:45.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:45.623+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:45.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:45.624+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:45.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:45.388+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:45.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:45.388+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:45.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:45.388+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:45.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:45.388+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:46.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:46.393+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:46.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:46.393+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:46.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:46.393+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:46.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:46.394+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:46.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:46.606+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:46.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:46.606+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:46.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:46.606+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:46.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:46.606+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:46.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:46.415+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:46.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:46.415+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:46.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:46.415+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:46.680 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:46.416+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:47.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:47.576+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:47.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:47.576+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:47.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:47.576+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:47.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:47.576+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:47.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:47.351+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:47.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:47.351+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:47.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:47.351+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:47.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:47.352+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:47.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:47.401+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:47.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:47.401+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:47.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:47.401+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:47.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:47.401+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:48.176 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:10:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:10:47.873+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 17588 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:10:48.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:48.302+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:48.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:48.302+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:48.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:48.302+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:48.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:48.303+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:48.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:48.536+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:48.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:48.536+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:48.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:48.536+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:48.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:48.537+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:48.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:48.417+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:48.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:48.417+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:48.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:48.417+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:48.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:48.418+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:49.572 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:49.277+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:49.572 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:49.277+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:49.572 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:49.277+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:49.572 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:49.277+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:49.572 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:49.459+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:49.573 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:49.459+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:49.573 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:49.459+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:49.573 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:49.459+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:49.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:49.585+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:49.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:49.585+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:49.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:49.585+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:49.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:49.585+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:50.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:50.251+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:50.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:50.251+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:50.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:50.251+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:50.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:50.252+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:50.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:50.546+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:50.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:50.546+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:50.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:50.546+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:50.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:50.546+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:50.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:50.441+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:50.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:50.441+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:50.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:50.441+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:50.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:50.441+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:51.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:51.448+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:51.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:51.448+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:51.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:51.448+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:51.676 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:51.448+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:51.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:51.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:51.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:51.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:51.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:51.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:51.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:51.501+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:51.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:51.274+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:51.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:51.274+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:51.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:51.274+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:51.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:51.274+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:52.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:52.250+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:52.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:52.250+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:52.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:52.250+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:52.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:52.250+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:52.863 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:52.453+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:52.863 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:52.453+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:52.863 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:52.453+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:52.863 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:52.453+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:52.864 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:52.513+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:52.864 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:52.513+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:52.864 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:52.513+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:52.864 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:52.514+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:53.176 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:10:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:10:52.875+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 17676 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:10:53.546 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:53.254+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:53.547 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:53.254+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:53.547 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:53.254+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:53.547 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:53.255+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:53.547 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:53.433+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:53.547 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:53.433+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:53.547 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:53.433+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:53.548 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:53.433+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:53.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:53.560+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:53.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:53.560+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:53.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:53.560+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:53.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:53.560+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:54.577 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:54.269+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:54.578 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:54.269+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:54.578 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:54.269+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:54.578 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:54.269+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:54.578 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:54.459+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:54.578 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:54.459+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:54.579 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:54.459+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:54.579 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:54.459+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:54.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:54.590+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:54.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:54.590+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:54.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:54.590+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:54.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:54.591+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:55.546 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:55.221+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:55.546 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:55.221+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:55.547 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:55.222+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:55.547 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:55.222+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:55.547 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:55.471+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:55.547 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:55.471+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:55.547 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:55.471+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:55.547 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:55.471+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:55.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:55.559+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:55.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:55.559+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:55.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:55.559+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:55.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:55.559+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:56.554 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:56.458+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:56.554 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:56.458+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:56.554 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:56.458+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:56.554 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:56.458+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:56.555 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:56.267+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:56.555 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:56.267+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:56.555 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:56.267+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:56.555 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:56.268+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:56.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:56.567+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:56.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:56.567+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:56.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:56.567+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:56.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:56.567+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:57.537 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:57.507+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:57.537 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:57.507+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:57.537 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:57.507+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:57.537 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:57.507+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:57.537 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:57.285+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:57.538 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:57.285+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:57.538 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:57.285+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:57.538 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:57.285+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:57.865 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:57.550+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:57.865 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:57.550+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:57.866 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:57.550+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:57.866 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:57.550+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:58.176 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:10:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:10:57.878+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 17767 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:10:58.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:58.334+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:58.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:58.334+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:58.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:58.334+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:58.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:58.334+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:58.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:58.584+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:58.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:58.584+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:58.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:58.584+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:58.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:58.584+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:58.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:58.522+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:58.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:58.522+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:58.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:58.522+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:58.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:58.522+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:59.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:59.589+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:59.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:59.589+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:59.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:59.589+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:10:59.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:10:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:10:59.589+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:10:59.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:59.348+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:59.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:59.348+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:59.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:59.348+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:10:59.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:10:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:10:59.349+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:10:59.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:59.483+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:59.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:59.483+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:59.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:59.483+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:10:59.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:10:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:10:59.483+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:00.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:00.326+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:00.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:00.326+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:00.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:00.326+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:00.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:00.327+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:00.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:00.638+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:00.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:00.638+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:00.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:00.638+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:00.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:00.639+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:00.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:00.444+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:00.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:00.444+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:00.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:00.444+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:00.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:00.445+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:01.617 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:01.487+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:01.618 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:01.487+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:01.618 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:01.487+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:01.618 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:01.488+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:01.618 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:01.328+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:01.618 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:01.328+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:01.619 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:01.328+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:01.619 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:01.329+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:01.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:01.631+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:01.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:01.631+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:01.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:01.631+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:01.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:01.631+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:02.623 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:02.530+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:02.623 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:02.530+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:02.623 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:02.530+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:02.623 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:02.530+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:02.624 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:02.344+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:02.624 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:02.344+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:02.624 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:02.344+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:02.624 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:02.345+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:02.878 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:11:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:11:02.881+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 17856 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:11:02.878 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:02.636+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:02.879 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:02.636+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:02.879 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:02.636+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:02.879 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:02.636+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:03.589 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:03.529+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:03.589 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:03.529+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:03.589 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:03.529+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:03.589 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:03.529+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:03.590 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:03.316+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:03.590 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:03.316+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:03.590 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:03.316+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:03.590 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:03.316+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:03.851 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:03.625+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:03.851 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:03.625+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:03.851 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:03.625+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:03.852 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:03.626+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:04.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:04.320+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:04.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:04.320+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:04.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:04.320+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:04.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:04.320+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:04.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:04.527+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:04.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:04.527+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:04.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:04.527+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:04.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:04.527+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:04.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:04.579+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:04.928 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:04.579+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:04.928 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:04.579+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:04.928 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:04.579+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:05.615 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:05.478+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:05.615 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:05.478+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:05.615 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:05.478+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:05.615 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:05.478+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:05.616 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:05.296+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:05.616 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:05.296+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:05.616 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:05.296+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:05.616 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:05.297+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:05.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:05.627+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:05.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:05.627+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:05.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:05.627+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:05.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:05.628+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:06.580 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:06.267+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:06.581 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:06.267+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:06.581 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:06.267+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:06.581 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:06.267+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:06.581 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:06.491+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:06.581 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:06.491+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:06.582 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:06.491+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:06.582 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:06.491+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:06.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:06.593+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:06.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:06.593+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:06.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:06.593+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:06.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:06.594+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:07.600 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:07.509+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:07.601 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:07.509+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:07.601 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:07.509+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:07.601 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:07.509+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:07.601 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:07.263+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:07.601 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:07.263+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:07.601 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:07.263+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:07.601 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:07.263+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:07.871 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:07.613+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:07.872 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:07.613+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:07.872 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:07.613+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:07.872 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:07.613+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:08.176 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:11:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:11:07.884+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 17947 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:11:08.571 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:08.290+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:08.571 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:08.290+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:08.571 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:08.290+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:08.572 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:08.290+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:08.572 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:08.486+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:08.572 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:08.486+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:08.572 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:08.486+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:08.572 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:08.486+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:08.852 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:08.584+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:08.852 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:08.584+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:08.852 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:08.584+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:08.852 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:08.584+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:09.498 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:09.260+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:09.499 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:09.260+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:09.499 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:09.260+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:09.499 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:09.260+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:09.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:09.548+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:09.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:09.548+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:09.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:09.548+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:09.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:09.548+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:09.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:09.511+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:09.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:09.511+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:09.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:09.511+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:09.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:09.512+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:10.426 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:10.270+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:10.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:10.270+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:10.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:10.270+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:10.428 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:10.270+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:10.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:10.546+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:10.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:10.546+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:10.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:10.546+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:10.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:10.546+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:10.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:10.510+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:10.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:10.510+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:10.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:10.510+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:10.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:10.511+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:11.676 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:11.521+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:11.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:11.521+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:11.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:11.521+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:11.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:11.521+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:11.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:11.287+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:11.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:11.287+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:11.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:11.287+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:11.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:11.287+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:11.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:11.476+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:11.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:11.476+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:11.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:11.476+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:11.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:11.476+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:12.525 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:12.270+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:12.526 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:12.270+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:12.526 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:12.270+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:12.526 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:12.270+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:12.526 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:12.450+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:12.527 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:12.450+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:12.527 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:12.450+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:12.527 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:12.451+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:12.874 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:12.538+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:12.875 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:12.538+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:12.875 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:12.538+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:12.875 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:12.538+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:13.176 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:11:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:11:12.887+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 18035 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:11:13.560 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:13.256+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:13.560 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:13.256+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:13.560 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:13.256+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:13.560 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:13.256+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:13.561 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:13.494+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:13.561 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:13.494+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:13.561 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:13.494+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:13.561 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:13.494+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:13.897 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:13.572+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:13.897 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:13.572+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:13.897 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:13.572+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:13.898 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:13.573+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:14.676 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:14.277+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:14.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:14.277+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:14.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:14.277+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:14.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:14.278+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:14.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:14.491+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:14.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:14.491+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:14.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:14.491+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:14.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:14.492+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:14.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:14.525+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:14.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:14.525+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:14.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:14.525+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:14.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:14.525+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:15.533 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:15.259+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:15.533 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:15.259+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:15.534 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:15.259+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:15.534 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:15.259+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:15.534 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:15.508+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:15.534 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:15.508+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:15.534 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:15.508+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:15.534 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:15.508+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:15.815 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:15.543+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:15.815 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:15.543+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:15.816 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:15.543+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:15.816 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:15.544+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:16.548 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:16.275+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:16.548 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:16.275+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:16.549 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:16.275+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:16.549 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:16.276+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:16.549 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:16.498+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:16.549 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:16.498+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:16.549 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:16.498+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:16.549 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:16.499+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:16.550 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:16.514+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:16.553 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:16.514+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:16.554 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:16.514+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:16.554 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:16.514+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:17.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:17.516+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:17.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:17.516+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:17.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:17.516+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:17.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:17.516+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:17.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:17.287+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:17.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:17.287+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:17.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:17.287+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:17.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:17.287+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:17.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:17.483+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:17.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:17.483+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:17.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:17.483+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:17.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:17.483+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:17.999 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:11:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:11:17.889+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 18126 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:11:18.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:18.560+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:18.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:18.560+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:18.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:18.560+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:18.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:18.560+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:18.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:18.312+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:18.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:18.312+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:18.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:18.312+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:18.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:18.312+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:18.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:18.498+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:18.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:18.498+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:18.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:18.498+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:18.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:18.498+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:19.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:19.569+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:19.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:19.569+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:19.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:19.569+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:19.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:19.569+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:19.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:19.327+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:19.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:19.327+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:19.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:19.327+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:19.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:19.327+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:19.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:19.491+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:19.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:19.491+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:19.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:19.491+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:19.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:19.492+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:20.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:20.332+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:20.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:20.332+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:20.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:20.332+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:20.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:20.332+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:20.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:20.575+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:20.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:20.575+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:20.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:20.575+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:20.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:20.576+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:20.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:20.459+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:20.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:20.459+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:20.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:20.459+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:20.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:20.459+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:21.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:21.557+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:21.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:21.557+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:21.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:21.557+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:21.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:21.557+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:21.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:21.312+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:21.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:21.312+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:21.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:21.312+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:21.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:21.312+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:21.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:21.437+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:21.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:21.437+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:21.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:21.437+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:21.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:21.437+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:22.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:22.557+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:22.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:22.557+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:22.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:22.557+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:22.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:22.557+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:22.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:22.320+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:22.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:22.320+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:22.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:22.320+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:22.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:22.320+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:22.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:22.413+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:22.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:22.413+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:22.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:22.413+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:22.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:22.413+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:23.177 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:11:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:11:22.892+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 18216 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:11:23.590 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:23.426+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:23.591 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:23.426+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:23.591 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:23.426+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:23.591 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:23.426+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:23.591 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:23.347+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:23.592 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:23.347+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:23.592 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:23.347+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:23.592 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:23.347+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:23.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:23.606+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:23.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:23.606+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:23.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:23.606+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:23.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:23.606+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:24.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:24.427+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:24.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:24.427+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:24.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:24.427+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:24.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:24.427+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:24.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:24.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:24.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:24.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:24.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:24.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:24.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:24.568+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:24.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:24.320+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:24.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:24.320+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:24.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:24.320+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:24.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:24.320+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:25.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:25.392+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:25.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:25.392+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:25.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:25.392+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:25.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:25.392+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:25.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:25.564+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:25.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:25.564+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:25.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:25.564+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:25.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:25.564+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:25.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:25.345+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:25.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:25.345+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:25.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:25.345+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:25.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:25.345+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:26.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:26.404+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:26.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:26.404+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:26.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:26.404+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:26.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:26.405+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:26.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:26.572+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:26.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:26.572+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:26.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:26.572+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:26.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:26.572+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:26.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:26.359+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:26.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:26.359+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:26.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:26.359+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:26.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:26.359+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:27.539 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:27.424+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:27.539 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:27.424+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:27.539 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:27.424+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:27.539 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:27.424+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:27.540 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:27.389+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:27.540 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:27.389+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:27.540 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:27.389+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:27.540 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:27.389+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:27.804 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:27.551+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:27.804 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:27.551+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:27.804 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:27.551+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:27.804 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:27.551+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:28.177 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:11:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:11:27.894+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 18308 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:11:28.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:28.574+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:28.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:28.574+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:28.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:28.574+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:28.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:28.574+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:28.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:28.343+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:28.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:28.343+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:28.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:28.343+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:28.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:28.343+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:28.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:28.415+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:28.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:28.415+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:28.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:28.415+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:28.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:28.415+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:29.677 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:29.337+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:29.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:29.337+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:29.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:29.337+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:29.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:29.337+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:29.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:29.561+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:29.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:29.561+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:29.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:29.561+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:29.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:29.561+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:29.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:29.462+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:29.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:29.462+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:29.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:29.462+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:29.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:29.462+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:30.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:30.358+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:30.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:30.358+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:30.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:30.358+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:30.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:30.358+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:30.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:30.548+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:30.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:30.548+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:30.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:30.548+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:30.928 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:30.548+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:30.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:30.460+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:30.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:30.460+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:30.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:30.460+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:30.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:30.460+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:31.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:31.343+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:31.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:31.343+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:31.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:31.343+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:31.428 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:31.343+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:31.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:31.596+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:31.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:31.596+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:31.928 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:31.596+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:31.928 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:31.596+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:31.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:31.448+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:31.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:31.448+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:31.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:31.448+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:31.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:31.448+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:32.562 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:32.315+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:32.562 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:32.315+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:32.562 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:32.315+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:32.562 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:32.315+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:32.563 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:32.458+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:32.563 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:32.458+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:32.563 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:32.458+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:32.563 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:32.458+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:32.885 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:32.574+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:32.885 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:32.574+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:32.886 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:32.574+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:32.886 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:32.574+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:33.177 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:11:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:11:32.897+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 18395 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:11:33.592 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:33.489+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:33.592 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:33.489+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:33.592 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:33.489+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:33.592 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:33.489+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:33.592 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:33.527+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:33.593 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:33.527+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:33.593 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:33.527+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:33.593 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:33.528+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:33.593 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:33.295+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:33.593 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:33.295+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:33.593 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:33.295+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:33.594 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:33.295+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:34.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:34.323+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:34.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:34.323+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:34.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:34.323+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:34.428 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:34.323+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:34.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:34.499+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:34.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:34.499+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:34.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:34.499+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:34.928 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:34.500+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:34.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:34.462+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:34.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:34.462+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:34.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:34.462+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:34.928 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:34.462+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:35.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:35.506+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:35.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:35.506+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:35.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:35.506+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:35.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:35.506+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:35.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:35.499+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:35.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:35.499+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:35.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:35.499+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:35.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:35.499+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:35.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:35.319+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:35.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:35.319+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:35.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:35.319+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:35.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:35.319+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:36.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:36.509+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:36.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:36.509+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:36.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:36.509+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:36.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:36.509+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:36.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:36.491+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:36.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:36.491+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:36.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:36.491+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:36.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:36.491+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:36.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:36.290+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:36.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:36.290+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:36.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:36.290+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:36.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:36.290+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:37.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:37.478+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:37.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:37.478+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:37.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:37.478+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:37.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:37.478+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:37.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:37.470+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:37.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:37.470+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:37.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:37.470+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:37.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:37.470+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:37.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:37.315+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:37.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:37.315+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:37.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:37.315+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:37.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:37.316+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:38.177 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:11:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:11:37.900+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 18485 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:11:38.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:38.454+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:38.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:38.454+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:38.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:38.454+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:38.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:38.455+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:38.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:38.346+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:38.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:38.346+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:38.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:38.346+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:38.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:38.347+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:38.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:38.433+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:38.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:38.433+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:38.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:38.433+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:38.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:38.434+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:39.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:39.393+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:39.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:39.393+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:39.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:39.393+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:39.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:39.393+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:39.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:39.443+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:39.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:39.443+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:39.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:39.443+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:39.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:39.444+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:39.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:39.371+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:39.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:39.371+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:39.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:39.371+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:39.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:39.372+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:40.427 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:40.398+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:40.428 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:40.398+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:40.428 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:40.398+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:40.428 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:40.398+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:40.428 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:40.377+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:40.428 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:40.377+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:40.429 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:40.377+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:40.429 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:40.378+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:40.429 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:40.361+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:40.429 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:40.361+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:40.429 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:40.361+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:40.429 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:40.361+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:41.677 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:41.354+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:41.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:41.354+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:41.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:41.354+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:41.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:41.354+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:41.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:41.351+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:41.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:41.351+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:41.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:41.351+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:41.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:41.351+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:41.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:41.383+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:41.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:41.383+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:41.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:41.383+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:41.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:41.384+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:42.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:42.393+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:42.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:42.393+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:42.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:42.393+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:42.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:42.394+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:42.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:42.406+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:42.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:42.406+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:42.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:42.406+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:42.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:42.407+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:42.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:42.349+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:42.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:42.349+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:42.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:42.349+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:42.680 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:42.350+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:43.177 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:11:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:11:42.903+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 18573 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:11:43.592 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:43.367+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:43.593 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:43.367+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:43.593 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:43.367+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:43.593 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:43.367+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:43.593 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:43.434+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:43.593 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:43.434+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:43.594 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:43.434+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:43.594 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:43.434+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:43.594 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:43.329+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:43.594 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:43.329+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:43.594 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:43.329+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:43.594 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:43.329+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:44.427 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:44.332+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:44.428 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:44.332+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:44.428 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:44.332+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:44.428 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:44.332+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:44.428 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:44.405+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:44.428 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:44.405+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:44.429 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:44.405+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:44.429 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:44.405+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:44.429 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:44.320+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:44.429 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:44.320+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:44.429 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:44.320+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:44.429 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:44.321+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:45.677 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:45.305+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:45.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:45.305+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:45.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:45.305+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:45.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:45.306+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:45.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:45.418+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:45.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:45.418+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:45.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:45.418+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:45.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:45.418+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:45.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:45.289+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:45.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:45.289+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:45.680 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:45.289+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:45.680 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:45.289+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:46.536 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:46.261+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:46.536 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:46.261+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:46.536 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:46.261+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:46.537 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:46.261+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:46.537 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:46.310+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:46.537 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:46.310+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:46.537 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:46.310+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:46.537 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:46.310+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:46.538 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:46.434+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:46.538 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:46.434+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:46.538 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:46.434+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:46.538 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:46.434+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:47.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:47.273+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:47.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:47.273+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:47.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:47.273+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:47.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:47.274+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:47.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:47.446+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:47.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:47.446+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:47.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:47.446+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:47.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:47.447+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:47.680 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:47.234+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:47.680 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:47.234+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:47.680 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:47.234+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:47.680 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:47.234+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:48.177 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:11:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:11:47.906+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 18664 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:11:48.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:48.263+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:48.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:48.263+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:48.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:48.263+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:48.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:48.263+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:48.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:48.405+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:48.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:48.405+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:48.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:48.405+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:48.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:48.405+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:48.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:48.238+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:48.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:48.238+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:48.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:48.238+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:48.680 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:48.238+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:49.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:49.411+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:49.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:49.411+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:49.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:49.411+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:49.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:49.411+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:49.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:49.297+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:49.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:49.297+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:49.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:49.297+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:49.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:49.298+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:49.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:49.267+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:49.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:49.267+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:49.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:49.267+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:49.680 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:49.267+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:50.428 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:50.332+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:50.428 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:50.332+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:50.428 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:50.332+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:50.428 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:50.332+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:50.429 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:50.394+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:50.429 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:50.394+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:50.429 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:50.394+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:50.429 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:50.394+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:50.429 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:50.294+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:50.429 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:50.294+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:50.429 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:50.294+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:50.430 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:50.294+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:51.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:51.422+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:51.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:51.422+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:51.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:51.422+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:51.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:51.423+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:51.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:51.298+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:51.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:51.298+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:51.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:51.298+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:51.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:51.299+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:51.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:51.319+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:51.680 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:51.319+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:51.680 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:51.319+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:51.680 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:51.319+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:52.327 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:52.260+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:52.327 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:52.260+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:52.327 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:52.260+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:52.327 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:52.260+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:52.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:52.420+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:52.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:52.420+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:52.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:52.420+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:52.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:52.420+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:52.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:52.339+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:52.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:52.339+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:52.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:52.339+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:52.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:52.339+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:53.023 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:11:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:11:52.909+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 18752 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:11:53.594 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:53.302+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:53.594 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:53.302+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:53.594 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:53.302+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:53.594 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:53.302+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:53.595 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:53.466+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:53.595 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:53.466+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:53.595 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:53.466+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:53.595 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:53.466+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:53.595 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:53.295+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:53.596 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:53.295+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:53.596 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:53.295+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:53.596 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:53.295+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:54.490 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:54.301+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:54.491 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:54.301+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:54.491 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:54.301+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:54.491 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:54.301+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:54.491 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:54.349+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:54.491 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:54.349+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:54.491 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:54.349+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:54.492 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:54.350+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:54.928 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:54.502+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:54.928 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:54.502+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:54.928 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:54.502+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:54.928 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:54.502+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:55.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:55.273+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:55.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:55.273+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:55.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:55.273+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:55.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:55.273+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:55.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:55.334+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:55.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:55.334+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:55.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:55.334+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:55.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:55.334+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:55.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:55.505+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:55.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:55.505+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:55.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:55.505+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:55.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:55.505+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:56.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:56.385+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:56.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:56.385+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:56.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:56.385+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:56.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:56.385+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:56.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:56.529+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:56.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:56.529+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:56.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:56.529+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:56.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:56.530+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:56.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:56.314+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:56.680 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:56.314+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:56.680 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:56.314+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:56.680 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:56.314+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:57.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:57.381+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:57.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:57.381+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:57.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:57.381+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:57.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:57.381+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:57.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:57.551+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:57.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:57.551+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:57.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:57.551+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:57.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:57.551+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:57.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:57.363+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:57.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:57.363+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:57.680 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:57.363+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:57.680 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:57.364+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:58.178 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:11:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:11:57.912+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 18842 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:11:58.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:58.328+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:58.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:58.328+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:58.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:58.328+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:58.678 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:58.328+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:58.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:58.511+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:58.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:58.511+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:58.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:58.511+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:58.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:58.511+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:58.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:58.408+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:58.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:58.408+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:58.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:58.408+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:58.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:58.408+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:59.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:59.384+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:59.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:59.384+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:59.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:59.384+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:11:59.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:11:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:11:59.385+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:11:59.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:59.465+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:59.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:59.465+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:59.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:59.465+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:26.932088+0000 (oldest deadline 2024-04-06T03:01:46.932088+0000) 2024-04-06T03:11:59.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:11:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:11:59.466+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:11:59.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:59.329+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:59.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:59.329+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:59.680 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:59.329+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:11:59.680 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:11:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:11:59.329+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:00.428 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:00.413+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:12:00.428 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:00.413+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:12:00.428 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:00.413+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:12:00.428 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:00.413+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:00.429 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:00.281+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:12:00.429 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:00.281+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:12:00.429 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:00.281+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:12:00.429 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:00.281+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:00.928 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:00.450+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:01.488 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:01.234+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:12:01.489 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:01.234+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:12:01.489 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:01.234+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:12:01.489 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:01.234+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:01.489 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:01.408+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:12:01.489 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:01.408+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:12:01.489 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:01.408+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:12:01.490 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:01.409+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:01.896 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:01.500+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:02.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:02.475+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:02.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:02.385+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:12:02.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:02.385+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:12:02.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:02.385+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:12:02.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:02.385+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:02.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:02.246+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:12:02.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:02.246+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:12:02.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:02.246+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:12:02.679 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:02.246+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:03.178 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:12:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:12:02.915+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 18930 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:12:03.506 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:03.207+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:12:03.506 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:03.207+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:12:03.506 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:03.207+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:32.534317+0000 (oldest deadline 2024-04-06T03:01:52.534317+0000) 2024-04-06T03:12:03.506 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:03.207+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:03.507 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:03.388+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:12:03.507 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:03.388+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:12:03.507 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:03.388+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:12:03.507 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:03.388+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:03.928 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:03.517+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:04.539 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:04.199+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:04.539 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:04.425+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:12:04.539 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:04.425+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:12:04.539 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:04.425+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:12:04.539 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:04.425+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:04.928 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:04.550+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:05.428 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:05.150+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:05.928 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:05.559+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:05.928 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:05.441+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:12:05.928 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:05.441+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:12:05.928 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:05.441+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:01:38.108173+0000 (oldest deadline 2024-04-06T03:01:58.108173+0000) 2024-04-06T03:12:05.929 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:05.442+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:06.395 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:06.136+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:06.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:06.407+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:06.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:06.566+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:07.399 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:07.154+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:07.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:07.411+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:07.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:07.545+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:08.178 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:12:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:12:07.918+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 19016 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:12:08.178 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:08.164+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:08.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:08.418+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:08.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:08.539+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:09.428 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:09.388+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:09.428 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:09.163+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:09.928 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:09.564+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:10.428 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:10.370+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:10.428 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:10.160+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:10.928 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:10.608+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:11.397 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:11.160+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:11.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:11.409+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:11.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:11.601+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:12.428 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:12.203+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:12.910 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:12.578+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:12.911 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:12.449+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:13.178 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:12:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:12:12.922+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 19103 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:12:13.595 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:13.428+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:13.595 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:13.197+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:13.928 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:13.607+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:14.395 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:14.154+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:14.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:14.407+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:14.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:14.603+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:15.387 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:15.145+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:15.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:15.399+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:15.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:15.633+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:16.428 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:16.397+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:16.429 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:16.154+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:16.928 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:16.647+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:17.428 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:17.358+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:17.428 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:17.117+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:17.914 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:17.638+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:18.178 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:12:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:12:17.925+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 19192 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:12:18.178 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:18.111+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:18.593 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:18.331+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:18.928 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:18.603+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:19.428 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:19.290+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:19.428 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:19.075+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:19.928 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:19.596+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:20.278 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:20.027+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:20.599 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:20.288+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:20.928 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:20.610+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:21.301 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:21.027+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:21.630 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:21.312+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:21.928 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:21.641+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:21.929 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:21.641+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:21.929 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:21.641+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:21.929 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:21.641+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:22.291 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:21.988+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:22.658 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:22.302+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:22.917 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:22.669+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:22.917 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:22.669+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:22.918 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:22.669+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:22.918 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:22.669+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:23.178 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:12:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:12:22.928+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 19281 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:12:23.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:23.016+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:23.596 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:23.329+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:23.928 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:23.636+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:23.929 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:23.636+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:23.929 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:23.636+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:23.929 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:23.636+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:24.346 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:24.012+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:24.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:24.357+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:24.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:24.598+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:24.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:24.598+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:24.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:24.598+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:24.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:24.598+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:25.331 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:25.054+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:25.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:25.343+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:25.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:25.553+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:25.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:25.553+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:25.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:25.553+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:25.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:25.554+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:26.340 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:26.029+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:26.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:26.351+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:26.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:26.539+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:26.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:26.539+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:26.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:26.539+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:26.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:26.539+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:27.375 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:27.035+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:27.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:27.386+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:27.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:27.386+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:27.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:27.386+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:27.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:27.386+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:27.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:27.518+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:27.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:27.518+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:27.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:27.518+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:27.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:27.518+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:28.178 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:12:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:12:27.931+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 19369 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:12:28.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:27.990+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:28.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:27.990+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:28.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:27.990+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:28.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:27.990+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:28.678 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:28.358+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:28.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:28.358+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:28.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:28.358+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:28.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:28.358+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:28.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:28.532+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:28.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:28.532+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:28.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:28.532+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:28.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:28.533+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:29.328 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:28.985+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:29.329 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:28.985+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:29.329 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:28.985+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:29.329 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:28.985+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:29.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:29.508+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:29.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:29.508+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:29.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:29.508+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:29.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:29.508+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:29.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:29.339+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:29.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:29.339+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:29.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:29.339+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:29.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:29.340+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:30.322 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:29.971+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:30.323 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:29.971+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:30.323 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:29.972+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:30.323 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:29.972+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:30.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:30.463+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:30.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:30.463+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:30.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:30.463+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:30.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:30.464+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:30.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:30.333+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:30.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:30.333+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:30.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:30.333+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:30.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:30.333+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:31.178 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:30.923+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:31.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:30.923+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:31.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:30.923+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:31.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:30.923+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:31.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:31.472+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:31.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:31.472+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:31.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:31.472+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:31.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:31.473+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:31.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:31.360+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:31.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:31.360+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:31.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:31.360+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:31.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:31.360+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:32.178 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:31.901+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:32.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:31.901+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:32.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:31.901+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:32.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:31.901+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:32.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:32.505+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:32.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:32.505+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:32.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:32.505+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:32.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:32.506+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:32.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:32.382+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:32.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:32.382+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:32.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:32.382+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:32.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:32.382+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:33.178 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:12:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:12:32.934+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 19457 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:12:33.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:32.934+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:33.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:32.934+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:33.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:32.934+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:33.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:32.937+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:33.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:33.359+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:33.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:33.359+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:33.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:33.359+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:33.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:33.359+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:33.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:33.541+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:33.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:33.541+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:33.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:33.541+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:33.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:33.541+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:34.178 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:33.913+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:34.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:33.913+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:34.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:33.913+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:34.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:33.913+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:34.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:34.368+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:34.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:34.368+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:34.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:34.369+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:34.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:34.369+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:34.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:34.544+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:34.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:34.544+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:34.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:34.544+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:34.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:34.544+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:35.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:34.937+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:35.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:34.937+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:35.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:34.937+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:35.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:34.937+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:35.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:35.556+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:35.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:35.556+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:35.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:35.556+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:35.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:35.557+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:35.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:35.327+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:35.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:35.327+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:35.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:35.327+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:35.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:35.327+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:36.178 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:35.897+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:36.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:35.897+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:36.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:35.897+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:36.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:35.897+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:36.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:36.586+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:36.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:36.586+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:36.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:36.586+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:36.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:36.587+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:36.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:36.350+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:36.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:36.350+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:36.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:36.350+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:36.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:36.350+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:37.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:36.884+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:37.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:36.884+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:37.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:36.884+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:37.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:36.884+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:37.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:37.538+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:37.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:37.538+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:37.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:37.538+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:37.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:37.538+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:37.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:37.335+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:37.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:37.335+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:37.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:37.335+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:37.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:37.335+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:38.178 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:12:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:12:37.938+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 19545 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:12:38.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:37.917+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:38.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:37.917+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:38.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:37.917+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:38.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:37.918+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:38.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:38.535+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:38.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:38.535+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:38.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:38.535+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:38.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:38.536+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:38.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:38.368+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:38.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:38.368+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:38.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:38.368+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:38.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:38.368+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:39.377 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:38.948+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:39.377 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:38.948+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:39.378 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:38.948+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:39.378 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:38.948+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:39.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:39.498+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:39.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:39.498+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:39.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:39.498+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:39.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:39.499+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:39.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:39.388+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:39.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:39.388+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:39.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:39.388+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:39.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:39.388+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:40.163 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:39.911+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:40.164 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:39.911+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:40.164 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:39.911+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:40.164 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:39.911+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:40.429 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:40.374+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:40.429 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:40.374+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:40.429 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:40.374+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:40.429 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:40.374+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:40.870 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:40.532+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:40.870 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:40.532+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:40.870 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:40.532+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:40.871 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:40.532+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:41.178 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:40.881+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:41.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:40.881+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:41.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:40.881+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:41.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:40.881+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:41.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:41.491+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:41.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:41.491+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:41.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:41.491+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:41.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:41.492+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:41.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:41.373+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:41.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:41.373+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:41.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:41.373+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:41.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:41.373+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:42.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:41.874+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:42.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:41.874+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:42.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:41.874+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:42.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:41.874+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:42.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:42.422+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:42.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:42.422+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:42.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:42.422+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:42.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:42.422+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:42.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:42.506+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:42.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:42.506+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:42.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:42.506+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:42.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:42.506+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:43.179 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:12:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:12:42.941+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 19630 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:12:43.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:42.839+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:43.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:42.839+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:43.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:42.839+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:43.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:42.839+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:43.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:43.412+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:43.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:43.412+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:43.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:43.412+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:43.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:43.413+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:43.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:43.487+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:43.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:43.487+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:43.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:43.487+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:43.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:43.487+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:44.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:43.832+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:44.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:43.832+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:44.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:43.832+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:44.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:43.832+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:44.662 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:44.415+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:44.662 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:44.415+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:44.663 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:44.415+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:44.663 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:44.415+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:44.663 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:44.524+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:44.663 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:44.524+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:44.663 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:44.524+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:44.663 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:44.525+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:44.929 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:44.863+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:44.929 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:44.863+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:44.929 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:44.863+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:44.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:44.863+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:45.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:45.379+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:45.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:45.379+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:45.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:45.379+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:45.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:45.380+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:45.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:45.565+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:45.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:45.565+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:45.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:45.565+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:45.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:45.565+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:46.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:45.845+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:46.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:45.845+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:46.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:45.845+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:46.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:45.845+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:46.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:46.360+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:46.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:46.360+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:46.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:46.360+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:46.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:46.360+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:46.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:46.522+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:46.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:46.522+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:46.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:46.522+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:46.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:46.522+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:47.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:46.882+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:47.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:46.882+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:47.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:46.882+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:47.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:46.882+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:47.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:47.368+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:47.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:47.368+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:47.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:47.368+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:47.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:47.368+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:47.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:47.497+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:47.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:47.497+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:47.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:47.497+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:47.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:47.497+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:48.179 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:12:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:12:47.944+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 19719 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:12:48.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:47.867+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:48.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:47.867+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:48.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:47.867+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:48.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:47.867+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:48.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:48.371+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:48.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:48.371+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:48.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:48.371+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:48.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:48.371+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:48.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:48.490+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:48.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:48.490+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:48.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:48.490+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:48.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:48.490+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:49.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:48.900+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:49.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:48.900+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:49.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:48.900+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:49.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:48.900+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:49.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:49.402+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:49.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:49.402+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:49.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:49.402+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:49.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:49.402+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:49.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:49.468+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:49.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:49.468+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:49.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:49.468+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:49.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:49.468+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:50.165 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:49.910+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:50.165 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:49.910+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:50.166 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:49.910+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:50.166 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:49.910+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:50.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:50.438+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:50.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:50.438+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:50.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:50.438+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:50.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:50.438+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:50.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:50.454+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:50.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:50.454+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:50.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:50.454+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:50.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:50.454+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:51.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:50.866+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:51.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:50.866+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:51.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:50.866+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:51.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:50.867+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:51.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:51.408+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:51.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:51.408+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:51.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:51.408+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:51.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:51.408+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:51.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:51.411+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:51.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:51.411+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:51.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:51.411+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:51.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:51.411+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:52.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:51.823+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:52.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:51.823+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:52.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:51.823+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:52.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:51.823+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:52.430 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:52.424+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:52.430 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:52.424+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:52.431 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:52.424+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:52.431 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:52.424+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:52.790 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:52.440+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:52.791 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:52.440+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:52.791 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:52.440+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:52.791 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:52.441+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:53.062 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:12:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:12:52.948+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 19808 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:12:53.062 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:52.801+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:53.063 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:52.801+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:53.063 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:52.801+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:53.063 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:52.801+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:53.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:53.407+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:53.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:53.407+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:53.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:53.407+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:53.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:53.407+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:53.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:53.463+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:53.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:53.463+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:53.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:53.463+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:53.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:53.463+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:54.122 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:53.781+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:54.122 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:53.781+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:54.122 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:53.781+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:54.122 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:53.781+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:54.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:54.434+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:54.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:54.434+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:54.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:54.434+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:54.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:54.434+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:54.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:54.413+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:54.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:54.413+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:54.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:54.413+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:54.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:54.413+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:55.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:54.738+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:55.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:54.738+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:55.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:54.738+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:55.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:54.739+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:55.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:55.418+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:55.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:55.418+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:55.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:55.418+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:55.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:55.418+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:55.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:55.367+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:55.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:55.367+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:55.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:55.367+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:55.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:55.368+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:56.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:55.746+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:56.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:55.746+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:56.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:55.746+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:56.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:55.746+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:56.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:56.417+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:56.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:56.417+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:56.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:56.417+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:56.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:56.417+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:56.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:56.401+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:56.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:56.401+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:56.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:56.401+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:56.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:56.401+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:57.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:56.702+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:57.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:56.702+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:57.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:56.702+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:57.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:56.702+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:57.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:57.380+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:57.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:57.380+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:57.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:57.380+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:57.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:57.381+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:57.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:57.416+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:57.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:57.416+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:57.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:57.416+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:57.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:57.416+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:57.952 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:12:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:12:57.951+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 19897 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:12:57.953 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:57.705+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:57.953 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:57.705+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:57.953 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:57.705+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:57.953 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:57.705+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:58.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:58.334+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:58.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:58.334+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:58.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:58.334+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:58.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:58.334+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:58.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:58.366+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:58.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:58.366+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:58.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:58.366+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:58.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:58.367+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:12:59.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:58.748+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:59.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:58.748+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:59.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:58.748+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:12:59.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:58.748+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:59.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:59.328+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:59.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:59.328+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:59.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:59.328+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:12:59.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:12:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:12:59.328+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:12:59.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:59.407+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:59.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:59.407+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:59.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:59.407+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:12:59.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:12:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:12:59.408+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:00.166 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:59.723+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:00.166 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:59.723+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:00.166 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:59.723+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:00.166 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:12:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:12:59.724+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:00.429 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:00.298+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:00.429 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:00.298+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:00.430 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:00.298+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:00.430 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:00.299+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:00.929 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:00.441+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:00.929 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:00.441+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:00.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:00.441+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:00.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:00.441+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:00.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:00.688+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:00.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:00.688+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:00.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:00.688+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:00.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:00.688+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:01.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:01.400+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:01.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:01.400+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:01.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:01.400+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:01.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:01.400+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:01.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:01.315+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:01.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:01.315+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:01.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:01.315+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:01.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:01.315+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:02.004 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:01.724+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:02.005 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:01.724+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:02.005 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:01.724+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:02.005 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:01.724+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:02.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:02.448+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:02.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:02.448+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:02.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:02.448+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:02.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:02.448+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:02.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:02.346+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:02.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:02.346+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:02.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:02.346+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:02.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:02.346+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:02.957 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:13:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:13:02.954+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 19985 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:13:02.958 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:02.713+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:02.958 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:02.713+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:02.958 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:02.713+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:02.958 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:02.714+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:03.598 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:03.345+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:03.599 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:03.345+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:03.599 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:03.345+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:03.599 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:03.345+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:03.599 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:03.424+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:03.599 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:03.424+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:03.600 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:03.424+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:03.600 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:03.425+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:03.929 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:03.709+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:03.929 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:03.709+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:03.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:03.709+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:03.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:03.709+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:04.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:04.435+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:04.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:04.435+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:04.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:04.435+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:04.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:04.435+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:04.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:04.346+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:04.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:04.346+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:04.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:04.346+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:04.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:04.346+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:05.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:04.745+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:05.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:04.745+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:05.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:04.745+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:05.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:04.745+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:05.679 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:05.431+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:05.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:05.431+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:05.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:05.431+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:05.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:05.431+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:05.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:05.374+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:05.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:05.374+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:05.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:05.374+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:05.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:05.374+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:06.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:05.702+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:06.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:05.702+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:06.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:05.702+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:06.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:05.702+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:06.671 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:06.482+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:06.671 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:06.482+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:06.671 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:06.482+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:06.671 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:06.482+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:06.671 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:06.327+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:06.672 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:06.327+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:06.672 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:06.327+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:06.672 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:06.327+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:06.929 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:06.681+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:06.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:06.681+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:06.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:06.681+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:06.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:06.681+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:07.651 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:07.442+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:07.651 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:07.442+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:07.651 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:07.442+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:07.651 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:07.442+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:07.652 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:07.321+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:07.652 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:07.321+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:07.652 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:07.321+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:07.652 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:07.321+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:07.929 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:07.661+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:07.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:07.661+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:07.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:07.661+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:07.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:07.661+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:08.296 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:13:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:13:07.957+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 20076 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:13:08.615 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:08.305+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:08.615 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:08.305+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:08.615 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:08.305+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:08.615 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:08.305+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:08.616 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:08.399+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:08.616 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:08.399+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:08.616 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:08.399+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:08.616 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:08.399+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:08.929 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:08.624+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:08.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:08.624+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:08.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:08.624+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:08.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:08.624+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:09.654 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:09.296+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:09.655 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:09.296+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:09.655 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:09.296+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:09.655 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:09.296+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:09.655 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:09.385+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:09.655 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:09.385+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:09.655 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:09.385+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:09.656 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:09.385+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:09.929 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:09.664+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:09.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:09.664+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:09.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:09.664+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:09.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:09.664+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:10.429 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:10.290+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:10.430 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:10.290+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:10.430 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:10.290+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:10.430 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:10.290+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:10.430 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:10.425+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:10.430 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:10.425+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:10.431 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:10.425+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:10.431 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:10.425+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:11.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:10.705+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:11.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:10.705+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:11.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:10.705+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:11.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:10.706+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:11.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:11.317+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:11.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:11.317+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:11.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:11.317+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:11.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:11.318+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:11.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:11.471+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:11.681 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:11.471+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:11.681 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:11.471+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:11.681 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:11.472+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:12.004 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:11.756+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:12.005 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:11.756+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:12.005 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:11.756+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:12.005 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:11.756+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:12.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:12.315+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:12.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:12.315+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:12.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:12.315+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:12.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:12.316+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:12.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:12.464+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:12.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:12.464+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:12.681 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:12.464+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:12.681 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:12.464+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:13.077 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:13:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:13:12.961+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 20166 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:13:13.077 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:12.763+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:13.078 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:12.763+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:13.078 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:12.763+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:13.078 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:12.763+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:13.429 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:13.315+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:13.430 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:13.315+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:13.430 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:13.315+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:13.430 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:13.315+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:13.792 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:13.460+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:13.793 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:13.460+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:13.793 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:13.460+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:13.793 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:13.460+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:14.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:13.802+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:14.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:13.802+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:14.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:13.802+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:14.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:13.802+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:14.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:14.323+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:14.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:14.323+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:14.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:14.324+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:14.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:14.324+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:14.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:14.489+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:14.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:14.489+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:14.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:14.489+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:14.681 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:14.489+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:15.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:14.804+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:15.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:14.804+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:15.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:14.804+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:15.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:14.804+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:15.626 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:15.526+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:15.627 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:15.526+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:15.627 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:15.526+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:15.627 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:15.526+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:15.627 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:15.339+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:15.627 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:15.339+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:15.627 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:15.339+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:15.627 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:15.339+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:15.929 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:15.837+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:15.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:15.837+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:15.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:15.837+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:15.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:15.838+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:16.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:16.306+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:16.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:16.306+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:16.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:16.306+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:16.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:16.307+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:16.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:16.521+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:16.681 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:16.521+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:16.681 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:16.521+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:16.681 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:16.521+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:17.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:16.887+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:17.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:16.887+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:17.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:16.887+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:17.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:16.887+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:17.557 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:17.564+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:17.558 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:17.564+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:17.559 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:17.315+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:17.559 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:17.315+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:17.559 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:17.315+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:17.559 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:17.315+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:17.830 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:17.564+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:17.830 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:17.565+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:18.179 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:13:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:13:17.964+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 20257 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:13:18.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:17.840+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:18.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:17.840+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:18.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:17.840+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:18.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:17.840+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:18.568 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:18.308+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:18.569 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:18.308+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:18.569 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:18.308+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:18.569 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:18.308+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:18.874 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:18.578+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:18.875 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:18.578+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:18.875 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:18.578+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:18.875 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:18.579+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:19.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:18.884+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:19.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:18.884+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:19.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:18.884+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:19.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:18.884+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:19.618 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:19.318+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:19.618 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:19.318+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:19.618 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:19.318+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:19.618 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:19.319+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:19.908 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:19.627+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:19.909 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:19.627+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:19.909 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:19.627+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:19.909 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:19.628+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:20.167 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:19.918+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:20.168 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:19.918+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:20.168 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:19.918+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:20.168 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:19.918+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:20.430 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:20.343+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:20.430 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:20.343+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:20.430 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:20.343+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:20.430 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:20.344+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:20.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:20.636+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:20.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:20.636+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:20.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:20.636+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:20.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:20.636+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:20.931 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:20.880+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:20.931 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:20.880+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:20.931 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:20.880+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:20.931 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:20.881+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:21.620 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:21.346+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:21.620 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:21.346+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:21.621 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:21.346+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:21.621 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:21.347+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:21.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:21.630+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:21.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:21.630+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:21.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:21.630+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:21.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:21.630+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:21.931 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:21.874+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:21.931 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:21.874+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:21.931 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:21.874+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:21.931 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:21.874+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:22.629 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:22.359+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:22.630 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:22.359+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:22.630 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:22.359+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:22.630 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:22.360+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:22.908 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:22.639+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:22.908 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:22.639+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:22.909 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:22.639+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:22.909 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:22.639+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:23.180 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:13:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:13:22.967+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 20348 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:13:23.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:22.918+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:23.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:22.918+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:23.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:22.918+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:23.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:22.918+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:23.657 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:23.400+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:23.658 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:23.400+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:23.658 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:23.400+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:23.658 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:23.400+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:23.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:23.869+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:23.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:23.869+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:23.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:23.869+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:23.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:23.869+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:23.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:23.667+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:23.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:23.667+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:23.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:23.667+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:23.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:23.667+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:24.613 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:24.372+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:24.613 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:24.372+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:24.613 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:24.372+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:24.614 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:24.373+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:24.929 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:24.622+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:24.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:24.622+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:24.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:24.622+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:24.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:24.623+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:24.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:24.869+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:24.931 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:24.869+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:24.931 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:24.869+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:24.931 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:24.870+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:25.639 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:25.335+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:25.640 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:25.335+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:25.640 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:25.335+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:25.640 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:25.335+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:25.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:25.649+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:25.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:25.649+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:25.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:25.649+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:25.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:25.649+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:25.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:25.838+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:25.931 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:25.838+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:25.931 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:25.838+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:25.931 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:25.838+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:26.641 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:26.340+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:26.641 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:26.340+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:26.641 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:26.340+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:26.641 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:26.341+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:26.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:26.886+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:26.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:26.886+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:26.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:26.886+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:26.930 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:26.886+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:26.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:26.651+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:26.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:26.651+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:26.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:26.651+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:26.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:26.651+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:27.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:27.386+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:27.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:27.386+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:27.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:27.386+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:27.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:27.387+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:27.681 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:27.615+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:27.681 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:27.615+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:27.681 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:27.615+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:27.681 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:27.615+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:28.180 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:13:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:13:27.970+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 20439 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:13:28.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:27.853+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:28.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:27.853+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:28.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:27.853+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:28.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:27.853+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:28.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:28.619+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:28.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:28.619+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:28.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:28.619+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:28.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:28.620+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:28.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:28.431+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:28.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:28.431+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:28.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:28.431+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:28.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:28.431+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:29.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:28.830+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:29.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:28.830+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:29.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:28.830+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:29.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:28.830+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:29.866 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:29.471+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:29.867 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:29.471+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:29.867 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:29.471+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:29.867 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:29.471+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:29.867 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:29.629+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:29.867 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:29.629+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:29.867 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:29.629+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:29.868 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:29.629+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:30.168 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:29.876+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:30.168 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:29.876+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:30.169 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:29.876+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:30.169 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:29.876+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:30.817 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:30.649+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:30.817 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:30.649+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:30.817 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:30.649+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:30.818 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:30.649+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:30.818 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:30.457+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:30.818 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:30.457+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:30.818 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:30.457+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:30.818 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:30.457+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:31.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:30.826+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:31.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:30.826+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:31.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:30.826+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:31.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:30.827+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:31.789 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:31.656+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:31.789 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:31.656+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:31.790 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:31.656+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:31.790 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:31.656+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:31.790 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:31.454+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:31.790 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:31.454+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:31.790 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:31.454+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:31.790 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:31.454+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:32.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:31.798+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:32.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:31.798+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:32.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:31.798+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:32.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:31.799+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:32.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:32.672+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:32.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:32.672+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:32.680 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:32.672+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:32.681 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:32.672+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:32.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:32.424+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:32.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:32.424+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:32.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:32.424+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:32.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:32.425+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:33.090 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:13:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:13:32.974+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 20529 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:13:33.090 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:32.758+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:33.090 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:32.758+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:33.091 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:32.758+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:33.091 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:32.758+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:33.723 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:33.462+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:33.723 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:33.462+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:33.723 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:33.462+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:33.723 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:33.462+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:33.724 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:33.667+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:33.724 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:33.667+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:33.724 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:33.667+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:33.724 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:33.667+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:34.059 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:33.732+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:34.059 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:33.732+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:34.059 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:33.732+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:34.059 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:33.732+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:34.747 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:34.462+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:34.747 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:34.462+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:34.747 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:34.462+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:34.747 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:34.462+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:34.747 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:34.657+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:34.748 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:34.657+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:34.748 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:34.657+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:34.748 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:34.657+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:35.095 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:34.756+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:35.096 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:34.756+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:35.096 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:34.756+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:35.096 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:34.756+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:35.745 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:35.473+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:35.746 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:35.473+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:35.746 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:35.473+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:35.746 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:35.473+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:35.746 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:35.622+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:35.746 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:35.622+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:35.747 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:35.622+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:35.747 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:35.622+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:36.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:35.754+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:36.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:35.754+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:36.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:35.754+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:36.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:35.755+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:36.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:36.589+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:36.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:36.589+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:36.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:36.589+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:36.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:36.590+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:36.931 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:36.717+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:36.931 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:36.717+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:36.931 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:36.717+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:36.931 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:36.717+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:36.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:36.499+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:36.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:36.499+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:36.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:36.499+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:36.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:36.499+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:37.930 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:37.506+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:37.930 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:37.506+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:37.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:37.506+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:37.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:37.506+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:37.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:37.626+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:37.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:37.626+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:37.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:37.626+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:37.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:37.626+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:37.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:37.698+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:37.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:37.698+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:37.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:37.698+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:37.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:37.698+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:38.430 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:13:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:13:37.977+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 20619 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:13:38.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:38.607+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:38.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:38.608+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:38.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:38.608+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:38.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:38.608+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:38.931 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:38.738+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:38.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:38.738+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:38.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:38.738+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:38.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:38.738+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:38.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:38.538+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:38.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:38.538+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:38.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:38.538+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:38.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:38.539+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:39.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:39.602+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:39.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:39.602+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:39.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:39.602+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:39.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:39.603+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:39.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:39.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:39.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:39.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:39.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:39.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:39.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:39.501+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:39.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:39.725+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:39.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:39.725+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:39.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:39.725+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:39.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:39.725+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:40.930 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:40.525+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:40.930 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:40.525+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:40.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:40.525+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:40.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:40.525+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:40.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:40.631+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:40.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:40.631+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:40.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:40.631+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:40.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:40.631+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:40.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:40.740+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:40.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:40.740+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:40.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:40.740+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:40.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:40.741+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:41.930 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:41.529+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:41.930 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:41.529+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:41.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:41.529+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:41.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:41.529+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:41.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:41.673+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:41.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:41.673+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:41.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:41.673+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:41.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:41.673+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:41.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:41.756+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:41.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:41.756+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:41.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:41.756+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:41.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:41.757+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:42.930 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:42.554+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:42.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:42.554+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:42.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:42.554+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:42.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:42.554+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:42.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:42.719+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:42.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:42.719+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:42.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:42.719+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:42.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:42.719+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:42.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:42.791+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:42.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:42.791+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:42.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:42.791+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:42.933 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:42.791+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:43.430 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:13:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:13:42.980+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 20709 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:13:43.754 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:43.507+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:43.754 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:43.507+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:43.754 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:43.507+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:43.754 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:43.507+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:43.754 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:43.742+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:43.755 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:43.742+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:43.755 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:43.742+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:43.755 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:43.742+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:44.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:43.763+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:44.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:43.763+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:44.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:43.763+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:44.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:43.763+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:44.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:44.703+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:44.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:44.703+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:44.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:44.703+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:44.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:44.703+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:44.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:44.557+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:44.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:44.557+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:44.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:44.557+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:44.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:44.557+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:44.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:44.749+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:44.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:44.749+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:44.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:44.749+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:44.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:44.749+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:45.930 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:45.545+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:45.930 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:45.545+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:45.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:45.545+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:45.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:45.545+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:45.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:45.741+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:45.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:45.741+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:45.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:45.741+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:45.931 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:45.741+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:45.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:45.783+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:45.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:45.783+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:45.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:45.783+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:45.932 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:45.783+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:46.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:46.566+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:46.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:46.566+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:46.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:46.566+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:46.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:46.566+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:47.180 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:46.719+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:47.180 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:46.719+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:47.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:46.719+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:47.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:46.719+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:47.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:46.771+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:47.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:46.771+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:47.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:46.771+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:47.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:46.771+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:47.808 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:47.543+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:47.809 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:47.543+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:47.809 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:47.543+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:47.809 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:47.544+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:47.809 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:47.714+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:47.809 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:47.714+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:47.810 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:47.714+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:47.810 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:47.715+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:48.099 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:13:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:13:47.983+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 20800 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:13:48.099 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:47.818+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:48.099 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:47.818+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:48.100 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:47.818+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:48.100 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:47.818+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:48.842 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:48.528+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:48.842 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:48.528+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:48.842 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:48.528+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:48.843 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:48.528+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:48.843 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:48.746+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:48.843 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:48.746+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:48.843 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:48.746+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:48.843 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:48.747+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:49.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:48.852+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:49.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:48.852+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:49.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:48.852+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:49.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:48.852+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:49.767 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:49.507+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:49.767 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:49.507+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:49.767 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:49.507+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:49.767 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:49.507+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:50.170 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:49.847+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:50.170 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:49.847+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:50.171 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:49.847+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:50.171 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:49.847+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:50.171 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:49.776+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:50.171 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:49.776+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:50.171 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:49.776+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:50.171 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:49.776+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:50.793 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:50.459+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:50.793 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:50.459+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:50.793 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:50.459+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:50.793 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:50.459+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:51.180 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:50.802+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:51.180 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:50.802+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:51.180 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:50.802+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:51.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:50.802+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:51.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:50.894+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:51.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:50.894+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:51.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:50.894+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:51.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:50.894+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:51.790 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:51.441+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:51.790 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:51.442+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:51.790 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:51.442+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:51.790 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:51.442+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:52.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:51.904+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:52.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:51.904+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:52.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:51.904+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:52.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:51.904+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:52.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:51.799+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:52.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:51.799+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:52.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:51.799+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:52.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:51.799+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:52.491 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:52.427+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:52.491 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:52.427+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:52.491 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:52.427+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:52.491 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:52.427+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:53.002 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:13:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:13:52.987+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 20889 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:13:53.002 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:52.756+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:53.003 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:52.756+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:53.003 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:52.756+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:53.003 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:52.756+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:53.003 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:52.855+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:53.003 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:52.855+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:53.003 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:52.855+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:53.003 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:52.856+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:53.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:53.420+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:53.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:53.420+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:53.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:53.420+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:53.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:53.420+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:54.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:53.905+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:54.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:53.905+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:54.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:53.905+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:54.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:53.905+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:54.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:53.792+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:54.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:53.792+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:54.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:53.792+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:54.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:53.792+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:54.830 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:54.467+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:54.830 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:54.467+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:54.831 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:54.467+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:54.831 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:54.467+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:55.180 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:54.838+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:55.180 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:54.839+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:55.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:54.839+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:55.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:54.839+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:55.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:54.929+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:55.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:54.929+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:55.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:54.929+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:55.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:54.929+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:55.563 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:55.445+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:55.563 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:55.445+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:55.564 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:55.445+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:55.564 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:55.445+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:56.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:55.954+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:56.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:55.954+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:56.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:55.954+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:56.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:55.954+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:56.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:55.887+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:56.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:55.887+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:56.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:55.887+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:56.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:55.887+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:56.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:56.403+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:56.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:56.403+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:56.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:56.403+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:56.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:56.403+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:57.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:56.999+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:57.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:56.999+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:57.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:56.999+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:57.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:56.999+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:57.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:56.907+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:57.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:56.907+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:57.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:56.907+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:57.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:56.907+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:57.915 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:57.451+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:57.915 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:57.451+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:57.915 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:57.451+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:57.915 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:57.451+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:58.180 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:13:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:13:57.989+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 20980 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:13:58.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:57.998+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:58.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:57.998+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:58.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:57.998+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:58.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:57.999+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:58.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:57.923+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:58.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:57.923+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:58.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:57.923+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:58.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:57.923+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:58.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:58.406+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:58.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:58.406+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:58.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:58.406+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:58.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:58.406+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:59.180 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:58.880+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:59.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:58.880+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:59.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:58.880+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:13:59.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:58.880+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:13:59.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:58.979+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:59.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:58.979+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:59.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:58.979+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:13:59.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:13:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:13:58.979+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:13:59.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:59.369+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:59.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:59.369+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:59.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:59.369+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:13:59.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:13:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:13:59.369+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:00.171 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:59.882+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:00.172 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:59.882+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:00.172 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:59.882+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:00.172 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:13:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:13:59.882+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:00.172 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:00.021+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:00.172 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:00.021+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:00.172 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:00.021+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:00.172 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:00.022+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:00.430 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:00.356+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:00.431 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:00.356+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:00.431 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:00.356+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:00.431 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:00.356+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:01.180 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:00.914+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:01.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:00.914+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:01.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:00.914+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:01.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:00.914+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:01.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:00.976+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:01.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:00.976+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:01.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:00.976+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:01.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:00.976+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:01.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:01.382+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:01.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:01.382+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:01.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:01.382+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:01.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:01.383+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:01.979 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:01.902+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:01.979 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:01.902+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:01.979 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:01.902+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:01.979 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:01.903+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:02.359 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:01.988+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:02.359 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:01.988+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:02.360 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:01.988+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:02.360 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:01.988+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:02.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:02.368+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:02.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:02.368+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:02.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:02.368+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:02.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:02.369+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:03.180 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:14:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:14:02.993+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 21068 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:14:03.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:02.977+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:03.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:02.977+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:03.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:02.977+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:03.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:02.977+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:03.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:02.917+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:03.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:02.917+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:03.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:02.917+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:03.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:02.917+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:03.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:03.418+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:03.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:03.418+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:03.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:03.418+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:03.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:03.419+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:04.180 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:04.018+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:04.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:04.018+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:04.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:04.018+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:04.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:04.018+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:04.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:03.875+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:04.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:03.875+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:04.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:03.875+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:04.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:03.876+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:04.781 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:04.441+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:04.781 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:04.441+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:04.781 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:04.441+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:04.781 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:04.441+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:05.044 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:04.853+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:05.045 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:04.853+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:05.045 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:04.853+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:05.045 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:04.853+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:05.351 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:05.053+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:05.352 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:05.053+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:05.352 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:05.053+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:05.352 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:05.053+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:05.680 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:05.429+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:05.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:05.429+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:05.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:05.429+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:05.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:05.429+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:06.180 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:05.884+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:06.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:05.884+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:06.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:05.884+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:06.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:05.885+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:06.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:06.012+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:06.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:06.012+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:06.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:06.012+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:06.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:06.012+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:06.890 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:06.448+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:06.890 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:06.448+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:06.890 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:06.448+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:06.890 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:06.449+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:07.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:06.898+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:07.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:06.898+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:07.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:06.898+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:07.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:06.899+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:07.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:07.044+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:07.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:07.044+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:07.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:07.044+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:07.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:07.044+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:07.917 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:07.441+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:07.918 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:07.441+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:07.918 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:07.441+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:07.918 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:07.441+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:08.180 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:14:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:14:07.997+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 21150 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:14:08.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:07.926+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:08.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:07.926+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:08.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:07.926+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:08.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:07.926+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:08.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:08.087+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:08.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:08.087+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:08.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:08.087+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:08.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:08.088+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:08.871 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:08.479+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:08.871 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:08.479+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:08.871 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:08.479+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:08.871 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:08.480+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:09.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:08.880+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:09.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:08.880+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:09.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:08.880+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:09.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:08.880+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:09.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:09.039+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:09.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:09.039+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:09.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:09.039+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:09.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:09.039+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:09.871 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:09.491+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:09.871 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:09.491+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:09.871 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:09.491+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:09.871 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:09.491+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:10.173 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:10.042+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:10.173 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:10.042+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:10.173 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:10.042+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:10.173 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:10.042+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:10.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:09.879+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:10.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:09.879+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:10.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:09.879+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:10.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:09.880+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:10.902 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:10.493+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:10.902 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:10.493+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:10.903 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:10.493+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:10.903 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:10.493+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:11.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:10.911+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:11.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:10.911+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:11.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:10.911+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:11.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:10.911+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:11.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:11.052+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:11.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:11.052+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:11.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:11.052+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:11.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:11.052+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:11.904 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:11.536+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:11.905 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:11.536+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:11.905 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:11.536+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:11.905 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:11.537+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:12.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:12.092+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:12.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:12.092+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:12.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:12.092+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:12.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:12.092+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:12.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:11.912+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:12.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:11.912+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:12.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:11.912+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:12.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:11.913+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:12.929 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:12.578+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:12.929 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:12.578+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:12.929 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:12.578+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:12.930 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:12.578+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:13.181 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:14:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:14:13.000+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 21231 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:14:13.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:13.061+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:13.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:13.061+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:13.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:13.061+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:13.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:13.062+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:13.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:12.938+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:13.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:12.938+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:13.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:12.938+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:13.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:12.938+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:13.908 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:13.538+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:13.908 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:13.538+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:13.908 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:13.538+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:13.908 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:13.539+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:14.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:13.916+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:14.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:13.916+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:14.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:13.916+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:14.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:13.917+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:14.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:14.100+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:14.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:14.100+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:14.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:14.100+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:14.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:14.101+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:14.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:14.512+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:14.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:14.512+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:14.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:14.512+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:14.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:14.513+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:15.431 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:14.957+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:15.431 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:14.957+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:15.431 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:14.957+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:15.431 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:14.958+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:15.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:15.093+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:15.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:15.093+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:15.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:15.093+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:15.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:15.093+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:15.906 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:15.548+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:15.906 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:15.548+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:15.906 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:15.548+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:15.906 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:15.549+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:16.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:15.914+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:16.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:15.914+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:16.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:15.914+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:16.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:15.915+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:16.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:16.134+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:16.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:16.134+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:16.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:16.134+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:16.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:16.134+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:16.900 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:16.519+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:16.900 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:16.519+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:16.901 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:16.519+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:16.901 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:16.519+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:17.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:17.113+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:17.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:17.113+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:17.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:17.113+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:17.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:17.114+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:17.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:16.909+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:17.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:16.909+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:17.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:16.909+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:17.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:16.909+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:17.894 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:17.540+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:17.895 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:17.540+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:17.895 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:17.540+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:17.895 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:17.540+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:18.146 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:14:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:14:18.003+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 21313 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:14:18.147 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:17.903+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:18.147 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:17.903+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:18.147 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:17.903+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:18.147 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:17.903+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:18.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:18.155+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:18.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:18.155+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:18.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:18.155+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:18.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:18.155+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:18.884 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:18.578+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:18.884 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:18.578+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:18.884 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:18.578+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:18.884 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:18.579+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:19.166 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:18.892+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:19.167 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:18.892+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:19.167 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:18.892+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:19.167 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:18.893+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:19.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:19.175+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:19.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:19.175+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:19.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:19.175+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:19.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:19.175+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:19.922 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:19.587+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:19.923 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:19.587+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:19.923 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:19.587+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:19.923 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:19.587+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:20.173 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:20.165+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:20.173 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:20.165+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:20.174 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:20.165+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:20.174 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:20.166+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:20.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:19.931+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:20.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:19.931+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:20.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:19.931+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:20.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:19.931+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:20.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:20.570+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:20.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:20.570+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:20.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:20.570+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:20.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:20.570+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:21.431 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:20.964+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:21.431 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:20.964+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:21.431 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:20.964+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:21.432 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:20.964+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:21.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:21.147+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:21.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:21.147+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:21.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:21.147+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:21.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:21.147+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:21.919 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:21.592+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:21.920 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:21.592+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:21.920 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:21.592+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:21.920 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:21.593+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:22.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:21.928+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:22.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:21.928+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:22.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:21.928+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:22.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:21.928+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:22.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:22.185+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:22.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:22.185+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:22.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:22.185+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:22.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:22.185+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:22.930 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:22.545+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:22.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:22.545+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:22.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:22.545+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:22.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:22.545+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:23.181 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:14:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:14:23.007+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 21398 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:14:23.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:22.939+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:23.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:22.939+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:23.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:22.939+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:23.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:22.939+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:23.566 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:23.222+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:23.567 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:23.222+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:23.567 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:23.222+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:23.567 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:23.222+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:23.886 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:23.575+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:23.886 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:23.575+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:23.887 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:23.575+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:23.887 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:23.575+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:24.172 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:23.894+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:24.172 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:23.894+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:24.172 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:23.894+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:24.173 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:23.895+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:24.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:24.180+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:24.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:24.180+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:24.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:24.180+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:24.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:24.181+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:24.919 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:24.546+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:24.920 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:24.547+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:24.920 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:24.547+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:24.920 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:24.547+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:25.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:24.927+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:25.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:24.927+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:25.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:24.928+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:25.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:24.928+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:25.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:25.132+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:25.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:25.132+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:25.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:25.132+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:25.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:25.132+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:25.881 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:25.588+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:25.882 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:25.588+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:25.882 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:25.588+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:25.882 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:25.589+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:26.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:25.890+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:26.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:25.890+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:26.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:25.890+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:26.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:25.890+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:26.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:26.091+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:26.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:26.091+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:26.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:26.091+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:26.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:26.091+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:26.854 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:26.557+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:26.854 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:26.557+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:26.854 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:26.557+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:26.854 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:26.558+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:27.124 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:26.862+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:27.124 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:26.862+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:27.124 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:26.862+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:27.124 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:26.862+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:27.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:27.132+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:27.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:27.132+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:27.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:27.132+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:27.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:27.132+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:27.897 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:27.533+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:27.897 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:27.533+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:27.898 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:27.533+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:27.898 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:27.533+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:28.167 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:14:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:14:28.011+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 21486 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:14:28.167 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:27.906+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:28.167 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:27.906+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:28.168 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:27.906+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:28.168 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:27.906+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:28.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:28.176+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:28.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:28.176+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:28.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:28.176+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:28.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:28.176+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:28.899 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:28.516+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:28.899 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:28.516+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:28.899 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:28.516+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:28.899 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:28.516+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:29.162 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:28.907+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:29.162 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:28.907+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:29.162 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:28.907+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:29.162 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:28.907+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:29.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:29.170+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:29.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:29.170+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:29.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:29.170+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:29.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:29.170+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:29.851 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:29.549+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:29.851 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:29.549+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:29.851 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:29.549+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:29.852 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:29.549+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:30.145 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:29.860+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:30.145 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:29.860+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:30.146 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:29.860+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:30.146 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:29.860+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:30.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:30.154+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:30.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:30.154+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:30.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:30.154+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:30.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:30.154+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:30.887 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:30.547+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:30.887 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:30.547+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:30.887 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:30.547+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:30.887 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:30.548+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:31.173 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:30.895+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:31.173 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:30.895+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:31.173 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:30.895+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:31.173 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:30.895+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:31.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:31.182+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:31.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:31.182+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:31.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:31.182+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:31.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:31.182+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:31.875 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:31.558+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:31.875 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:31.558+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:31.875 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:31.558+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:31.875 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:31.558+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:32.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:31.884+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:32.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:31.884+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:32.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:31.884+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:32.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:31.884+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:32.599 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:32.215+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:32.599 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:32.215+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:32.599 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:32.215+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:32.599 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:32.215+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:32.880 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:32.607+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:32.880 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:32.607+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:32.881 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:32.607+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:32.881 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:32.607+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:33.131 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:14:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:14:33.014+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 21579 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:14:33.131 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:32.889+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:33.132 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:32.889+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:33.132 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:32.889+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:33.132 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:32.889+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:33.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:33.229+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:33.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:33.229+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:33.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:33.229+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:33.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:33.229+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:33.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:33.621+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:33.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:33.621+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:33.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:33.621+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:33.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:33.622+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:33.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:33.849+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:33.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:33.849+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:33.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:33.849+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:33.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:33.850+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:34.648 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:34.261+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:34.648 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:34.261+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:34.648 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:34.261+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:34.648 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:34.261+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:34.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:34.656+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:34.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:34.656+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:34.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:34.656+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:34.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:34.656+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:34.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:34.833+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:34.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:34.833+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:34.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:34.833+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:34.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:34.833+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:35.607 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:35.279+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:35.608 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:35.279+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:35.608 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:35.279+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:35.608 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:35.279+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:35.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:35.705+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:35.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:35.705+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:35.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:35.705+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:35.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:35.705+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:35.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:35.811+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:35.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:35.811+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:35.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:35.811+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:35.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:35.811+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:36.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:36.326+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:36.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:36.326+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:36.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:36.326+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:36.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:36.326+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:37.181 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:36.732+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:37.181 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:36.732+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:37.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:36.732+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:37.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:36.732+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:37.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:36.839+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:37.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:36.839+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:37.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:36.839+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:37.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:36.839+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:37.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:37.322+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:37.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:37.322+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:37.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:37.322+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:37.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:37.323+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:38.010 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:37.793+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:38.010 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:37.793+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:38.010 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:37.793+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:38.010 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:37.793+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:38.011 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:37.703+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:38.011 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:37.703+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:38.011 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:37.703+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:38.011 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:37.703+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:38.284 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:14:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:14:38.017+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 21671 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:14:38.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:38.293+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:38.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:38.293+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:38.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:38.293+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:38.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:38.293+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:39.181 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:38.727+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:39.181 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:38.727+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:39.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:38.727+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:39.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:38.727+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:39.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:38.812+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:39.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:38.812+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:39.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:38.812+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:39.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:38.813+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:39.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:39.297+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:39.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:39.297+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:39.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:39.297+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:39.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:39.297+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:40.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:39.747+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:40.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:39.747+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:40.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:39.747+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:40.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:39.747+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:40.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:39.822+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:40.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:39.822+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:40.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:39.822+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:40.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:39.822+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:40.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:40.253+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:40.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:40.253+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:40.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:40.253+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:40.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:40.253+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:40.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:40.705+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:40.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:40.705+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:40.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:40.705+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:40.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:40.705+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:40.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:40.862+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:40.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:40.862+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:40.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:40.862+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:40.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:40.862+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:41.644 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:41.218+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:41.644 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:41.218+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:41.645 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:41.218+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:41.645 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:41.219+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:41.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:41.712+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:41.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:41.712+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:41.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:41.712+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:41.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:41.712+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:41.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:41.871+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:41.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:41.871+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:41.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:41.871+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:41.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:41.871+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:42.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:42.259+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:42.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:42.259+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:42.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:42.259+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:42.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:42.259+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:43.013 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:42.883+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:43.014 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:42.883+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:43.014 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:42.883+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:43.014 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:42.883+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:43.014 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:42.703+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:43.014 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:42.703+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:43.015 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:42.703+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:43.015 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:42.703+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:43.431 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:14:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:14:43.021+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 21762 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:14:43.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:43.214+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:43.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:43.214+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:43.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:43.214+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:43.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:43.214+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:43.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:43.665+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:43.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:43.665+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:43.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:43.665+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:43.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:43.665+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:43.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:43.858+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:43.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:43.858+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:43.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:43.858+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:43.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:43.858+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:44.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:44.184+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:44.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:44.184+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:44.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:44.184+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:44.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:44.184+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:44.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:44.676+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:44.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:44.676+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:44.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:44.676+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:44.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:44.676+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:44.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:44.873+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:44.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:44.873+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:44.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:44.873+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:44.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:44.873+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:45.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:45.199+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:45.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:45.199+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:45.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:45.199+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:45.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:45.199+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:46.162 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:45.723+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:46.162 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:45.723+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:46.162 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:45.723+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:46.162 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:45.723+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:46.163 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:45.920+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:46.163 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:45.920+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:46.163 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:45.920+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:46.163 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:45.920+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:46.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:46.170+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:46.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:46.170+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:46.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:46.170+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:46.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:46.170+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:47.181 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:46.755+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:47.181 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:46.755+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:47.181 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:46.755+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:47.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:46.755+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:47.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:46.918+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:47.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:46.918+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:47.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:46.918+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:47.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:46.918+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:47.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:47.203+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:47.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:47.203+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:47.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:47.203+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:47.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:47.204+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:48.017 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:47.737+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:48.018 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:47.737+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:48.018 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:47.737+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:48.018 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:47.737+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:48.018 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:47.965+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:48.018 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:47.965+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:48.018 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:47.965+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:48.018 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:47.965+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:48.431 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:14:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:14:48.024+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 21853 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:14:48.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:48.209+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:48.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:48.209+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:48.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:48.209+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:48.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:48.209+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:49.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:48.733+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:49.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:48.733+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:49.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:48.733+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:49.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:48.733+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:49.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:48.919+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:49.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:48.919+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:49.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:48.919+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:49.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:48.919+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:49.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:49.215+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:49.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:49.215+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:49.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:49.215+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:49.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:49.216+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:50.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:49.970+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:50.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:49.970+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:50.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:49.970+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:50.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:49.970+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:50.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:49.740+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:50.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:49.740+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:50.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:49.740+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:50.176 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:49.741+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:50.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:50.183+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:50.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:50.183+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:50.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:50.183+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:50.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:50.183+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:51.141 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:50.741+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:51.141 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:50.741+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:51.142 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:50.741+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:51.142 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:50.742+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:51.142 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:50.945+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:51.142 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:50.945+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:51.142 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:50.945+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:51.142 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:50.946+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:51.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:51.149+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:51.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:51.149+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:51.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:51.149+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:51.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:51.149+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:52.181 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:51.768+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:52.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:51.768+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:52.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:51.768+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:52.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:51.768+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:52.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:51.981+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:52.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:51.981+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:52.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:51.981+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:52.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:51.981+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:52.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:52.195+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:52.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:52.195+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:52.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:52.195+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:52.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:52.195+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:53.022 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:52.951+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:53.022 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:52.951+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:53.022 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:52.951+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:53.022 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:52.951+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:53.023 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:52.764+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:53.023 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:52.764+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:53.023 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:52.764+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:53.023 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:52.764+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:53.431 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:14:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:14:53.029+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 21942 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:14:53.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:53.197+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:53.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:53.197+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:53.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:53.197+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:53.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:53.197+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:53.897 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:53.753+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:53.897 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:53.753+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:53.897 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:53.753+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:53.897 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:53.753+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:54.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:53.904+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:54.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:53.904+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:54.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:53.904+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:54.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:53.905+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:54.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:54.199+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:54.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:54.199+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:54.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:54.199+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:54.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:54.199+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:55.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:54.739+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:55.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:54.739+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:55.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:54.739+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:55.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:54.739+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:55.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:54.901+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:55.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:54.901+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:55.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:54.901+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:55.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:54.901+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:55.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:55.220+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:55.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:55.220+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:55.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:55.220+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:55.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:55.220+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:56.181 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:55.706+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:56.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:55.706+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:56.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:55.706+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:56.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:55.707+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:56.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:55.933+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:56.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:55.933+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:56.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:55.933+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:56.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:55.934+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:56.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:56.230+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:56.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:56.230+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:56.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:56.230+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:56.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:56.230+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:57.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:56.725+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:57.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:56.725+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:57.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:56.725+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:57.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:56.725+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:57.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:56.896+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:57.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:56.896+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:57.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:56.896+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:57.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:56.897+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:57.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:57.207+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:57.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:57.207+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:57.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:57.207+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:57.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:57.208+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:58.025 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:57.864+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:58.026 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:57.864+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:58.026 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:57.864+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:58.026 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:57.864+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:58.026 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:57.710+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:58.026 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:57.710+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:58.026 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:57.710+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:58.027 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:57.710+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:58.431 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:14:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:14:58.033+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 22033 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:14:58.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:58.244+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:58.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:58.244+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:58.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:58.244+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:58.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:58.244+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:59.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:58.749+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:59.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:58.749+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:59.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:58.749+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:14:59.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:58.750+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:14:59.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:58.901+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:59.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:58.901+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:59.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:58.901+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:14:59.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:58.902+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:14:59.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:59.253+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:59.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:59.253+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:59.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:59.253+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:14:59.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:14:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:14:59.253+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:00.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:59.941+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:00.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:59.941+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:00.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:59.941+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:00.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:14:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:14:59.941+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:00.177 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:59.750+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:00.177 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:59.750+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:00.177 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:59.750+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:00.177 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:14:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:14:59.751+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:00.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:00.302+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:00.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:00.302+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:00.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:00.302+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:00.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:00.302+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:00.972 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:00.725+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:00.972 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:00.725+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:00.973 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:00.725+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:00.973 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:00.725+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:01.284 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:00.980+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:01.284 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:00.980+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:01.284 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:00.980+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:01.284 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:00.980+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:01.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:01.292+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:01.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:01.292+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:01.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:01.292+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:01.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:01.292+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:02.181 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:01.765+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:02.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:01.765+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:02.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:01.765+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:02.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:01.765+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:02.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:01.954+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:02.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:01.954+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:02.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:01.954+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:02.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:01.954+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:02.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:02.307+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:02.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:02.307+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:02.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:02.307+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:02.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:02.307+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:03.043 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:15:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:15:03.036+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 22122 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:15:03.043 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:02.793+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:03.043 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:02.793+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:03.044 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:02.793+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:03.044 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:02.793+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:03.044 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:02.986+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:03.044 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:02.986+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:03.044 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:02.986+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:03.044 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:02.986+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:03.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:03.276+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:03.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:03.276+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:03.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:03.276+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:03.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:03.276+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:03.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:03.744+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:03.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:03.744+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:03.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:03.744+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:03.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:03.744+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:04.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:04.247+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:04.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:04.247+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:04.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:04.247+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:04.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:04.248+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:04.432 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:04.024+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:04.433 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:04.024+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:04.433 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:04.024+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:04.433 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:04.024+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:05.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:04.763+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:05.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:04.763+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:05.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:04.763+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:05.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:04.763+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:05.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:04.991+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:05.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:04.991+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:05.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:04.991+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:05.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:04.991+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:05.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:05.287+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:05.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:05.287+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:05.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:05.288+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:05.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:05.288+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:06.025 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:05.726+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:06.025 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:05.726+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:06.025 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:05.726+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:06.025 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:05.726+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:06.431 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:06.245+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:06.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:06.245+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:06.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:06.245+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:06.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:06.245+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:06.432 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:06.033+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:06.432 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:06.033+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:06.432 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:06.033+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:06.433 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:06.033+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:06.985 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:06.694+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:06.985 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:06.694+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:06.986 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:06.694+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:06.986 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:06.694+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:07.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:07.215+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:07.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:07.215+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:07.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:07.215+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:07.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:07.215+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:07.432 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:06.993+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:07.433 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:06.993+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:07.433 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:06.993+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:07.433 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:06.993+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:08.033 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:07.977+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:08.033 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:07.977+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:08.033 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:07.977+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:08.033 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:07.977+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:08.034 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:07.744+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:08.034 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:07.744+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:08.034 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:07.744+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:08.034 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:07.744+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:08.432 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:15:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:15:08.040+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 22213 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:15:08.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:08.233+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:08.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:08.233+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:08.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:08.233+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:08.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:08.233+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:09.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:09.008+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:09.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:09.008+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:09.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:09.008+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:09.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:09.008+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:09.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:08.794+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:09.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:08.794+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:09.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:08.794+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:09.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:08.794+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:09.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:09.268+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:09.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:09.268+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:09.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:09.268+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:09.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:09.268+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:10.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:09.972+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:10.178 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:09.972+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:10.178 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:09.972+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:10.178 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:09.973+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:10.178 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:09.805+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:10.179 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:09.805+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:10.179 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:09.805+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:10.179 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:09.805+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:10.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:10.318+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:10.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:10.318+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:10.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:10.318+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:10.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:10.318+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:11.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:10.763+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:11.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:10.763+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:11.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:10.763+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:11.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:10.763+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:11.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:10.928+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:11.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:10.928+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:11.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:10.928+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:11.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:10.928+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:11.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:11.344+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:11.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:11.344+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:11.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:11.344+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:11.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:11.344+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:12.007 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:11.738+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:12.007 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:11.738+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:12.007 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:11.738+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:12.008 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:11.738+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:12.008 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:11.918+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:12.008 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:11.918+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:12.008 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:11.918+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:12.008 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:11.918+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:12.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:12.387+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:12.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:12.387+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:12.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:12.387+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:12.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:12.388+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:12.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:12.696+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:12.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:12.696+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:12.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:12.696+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:12.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:12.696+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:13.413 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:15:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:15:13.044+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 22302 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:15:13.414 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:12.947+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:13.414 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:12.947+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:13.414 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:12.947+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:13.414 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:12.948+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:13.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:13.421+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:13.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:13.421+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:13.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:13.421+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:13.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:13.421+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:14.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:13.740+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:14.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:13.740+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:14.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:13.740+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:14.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:13.741+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:14.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:13.906+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:14.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:13.906+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:14.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:13.906+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:14.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:13.907+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:14.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:14.377+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:14.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:14.377+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:14.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:14.377+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:14.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:14.377+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:15.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:14.741+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:15.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:14.741+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:15.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:14.741+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:15.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:14.742+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:15.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:14.945+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:15.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:14.945+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:15.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:14.945+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:15.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:14.946+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:15.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:15.331+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:15.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:15.331+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:15.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:15.331+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:15.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:15.331+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:15.949 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:15.698+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:15.950 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:15.698+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:15.950 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:15.698+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:15.950 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:15.699+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:15.950 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:15.941+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:15.950 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:15.941+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:15.951 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:15.941+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:15.951 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:15.941+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:16.646 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:16.341+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:16.646 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:16.341+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:16.646 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:16.341+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:16.646 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:16.342+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:16.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:16.653+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:16.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:16.653+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:16.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:16.653+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:16.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:16.653+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:17.354 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:16.978+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:17.354 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:16.978+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:17.354 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:16.978+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:17.354 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:16.978+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:17.625 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:17.362+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:17.625 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:17.362+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:17.625 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:17.362+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:17.625 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:17.362+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:17.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:17.632+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:17.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:17.632+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:17.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:17.632+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:17.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:17.632+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:18.335 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:15:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:15:18.048+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 22393 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:15:18.335 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:17.954+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:18.336 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:17.954+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:18.336 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:17.954+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:18.336 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:17.954+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:18.617 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:18.343+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:18.618 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:18.343+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:18.618 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:18.343+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:18.618 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:18.343+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:18.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:18.625+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:18.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:18.625+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:18.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:18.625+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:18.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:18.625+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:19.307 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:18.981+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:19.307 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:18.981+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:19.308 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:18.981+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:19.308 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:18.981+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:19.592 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:19.314+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:19.592 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:19.314+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:19.592 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:19.314+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:19.592 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:19.315+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:19.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:19.600+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:19.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:19.600+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:19.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:19.600+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:19.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:19.600+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:20.262 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:19.953+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:20.262 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:19.953+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:20.262 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:19.953+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:20.262 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:19.954+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:20.621 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:20.269+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:20.621 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:20.269+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:20.622 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:20.269+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:20.622 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:20.269+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:20.921 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:20.629+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:20.921 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:20.629+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:20.921 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:20.629+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:20.921 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:20.629+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:21.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:20.929+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:21.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:20.929+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:21.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:20.929+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:21.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:20.929+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:21.581 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:21.257+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:21.581 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:21.257+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:21.581 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:21.257+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:21.581 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:21.257+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:21.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:21.610+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:21.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:21.610+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:21.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:21.610+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:21.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:21.610+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:22.295 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:21.973+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:22.296 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:21.973+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:22.296 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:21.973+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:22.296 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:21.973+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:22.602 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:22.303+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:22.602 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:22.303+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:22.603 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:22.303+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:22.603 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:22.303+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:22.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:22.610+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:22.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:22.610+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:22.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:22.610+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:22.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:22.610+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:23.339 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:15:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:15:23.051+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 22484 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:15:23.339 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:23.000+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:23.339 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:23.000+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:23.340 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:23.000+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:23.340 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:23.000+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:23.611 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:23.347+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:23.611 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:23.347+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:23.611 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:23.347+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:23.611 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:23.347+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:23.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:23.646+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:23.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:23.646+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:23.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:23.646+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:23.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:23.646+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:24.367 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:23.981+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:24.367 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:23.981+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:24.367 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:23.981+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:24.367 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:23.981+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:24.653 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:24.375+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:24.653 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:24.375+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:24.653 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:24.375+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:24.653 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:24.375+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:24.654 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:24.605+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:24.654 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:24.605+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:24.654 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:24.605+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:24.654 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:24.605+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:25.331 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:25.015+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:25.332 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:25.015+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:25.332 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:25.015+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:25.332 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:25.015+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:25.682 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:25.559+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:25.682 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:25.559+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:25.682 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:25.559+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:25.682 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:25.559+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:25.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:25.339+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:25.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:25.339+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:25.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:25.339+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:25.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:25.339+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:26.284 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:25.969+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:26.284 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:25.969+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:26.284 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:25.969+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:26.284 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:25.969+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:26.560 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:26.291+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:26.561 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:26.291+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:26.561 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:26.291+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:26.561 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:26.291+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:26.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:26.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:26.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:26.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:26.926 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:26.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:26.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:26.568+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:27.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:26.933+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:27.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:26.933+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:27.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:26.933+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:27.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:26.934+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:27.599 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:27.308+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:27.599 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:27.308+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:27.599 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:27.308+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:27.600 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:27.308+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:27.882 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:27.606+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:27.883 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:27.606+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:27.883 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:27.606+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:27.883 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:27.607+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:28.173 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:15:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:15:28.055+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 22575 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:15:28.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:27.890+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:28.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:27.890+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:28.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:27.890+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:28.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:27.890+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:28.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:28.275+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:28.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:28.275+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:28.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:28.275+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:28.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:28.275+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:28.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:28.596+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:28.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:28.596+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:28.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:28.596+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:28.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:28.596+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:28.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:28.841+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:28.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:28.842+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:28.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:28.842+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:28.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:28.842+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:29.579 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:29.266+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:29.579 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:29.266+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:29.579 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:29.266+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:29.579 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:29.266+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:29.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:29.587+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:29.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:29.587+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:29.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:29.587+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:29.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:29.587+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:29.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:29.795+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:29.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:29.795+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:29.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:29.795+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:29.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:29.795+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:30.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:30.222+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:30.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:30.222+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:30.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:30.222+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:30.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:30.222+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:30.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:30.580+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:30.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:30.580+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:30.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:30.580+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:30.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:30.580+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:30.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:30.776+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:30.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:30.776+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:30.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:30.776+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:30.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:30.776+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:31.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:31.206+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:31.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:31.206+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:31.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:31.206+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:31.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:31.206+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:31.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:31.562+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:31.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:31.562+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:31.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:31.562+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:31.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:31.563+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:31.935 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:31.784+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:31.935 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:31.784+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:31.935 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:31.784+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:31.935 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:31.784+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:32.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:32.170+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:32.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:32.170+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:32.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:32.170+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:32.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:32.170+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:32.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:32.581+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:32.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:32.581+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:32.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:32.581+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:32.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:32.581+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:32.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:32.778+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:32.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:32.778+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:32.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:32.778+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:32.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:32.778+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:33.432 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:15:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:15:33.059+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 22664 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:15:33.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:33.131+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:33.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:33.131+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:33.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:33.131+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:33.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:33.131+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:33.869 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:33.558+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:33.869 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:33.558+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:33.869 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:33.558+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:33.870 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:33.558+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:33.870 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:33.768+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:33.870 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:33.768+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:33.870 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:33.768+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:33.870 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:33.768+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:34.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:34.099+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:34.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:34.099+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:34.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:34.099+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:34.183 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:34.099+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:34.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:34.603+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:34.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:34.603+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:34.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:34.603+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:34.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:34.603+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:34.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:34.773+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:34.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:34.773+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:34.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:34.773+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:34.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:34.774+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:35.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:35.141+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:35.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:35.141+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:35.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:35.141+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:35.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:35.141+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:35.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:35.590+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:35.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:35.590+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:35.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:35.590+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:35.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:35.590+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:35.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:35.798+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:35.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:35.798+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:35.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:35.798+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:35.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:35.799+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:36.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:36.147+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:36.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:36.147+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:36.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:36.147+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:36.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:36.147+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:36.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:36.636+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:36.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:36.636+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:36.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:36.636+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:36.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:36.636+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:36.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:36.785+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:36.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:36.786+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:36.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:36.786+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:36.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:36.786+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:37.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:37.154+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:37.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:37.154+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:37.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:37.154+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:37.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:37.154+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:37.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:37.657+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:37.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:37.657+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:37.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:37.657+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:37.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:37.658+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:37.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:37.773+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:37.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:37.773+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:37.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:37.773+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:37.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:37.773+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:38.432 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:15:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:15:38.062+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 22756 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:15:38.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:38.114+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:38.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:38.114+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:38.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:38.114+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:38.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:38.114+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:39.145 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:38.733+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:39.146 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:38.733+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:39.146 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:38.733+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:39.146 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:38.733+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:39.146 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:38.690+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:39.146 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:38.690+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:39.146 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:38.690+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:39.147 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:38.690+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:39.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:39.152+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:39.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:39.152+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:39.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:39.152+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:39.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:39.152+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:39.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:39.671+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:39.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:39.671+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:39.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:39.671+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:39.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:39.671+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:39.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:39.756+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:39.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:39.756+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:39.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:39.756+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:39.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:39.756+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:40.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:40.152+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:40.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:40.152+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:40.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:40.152+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:40.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:40.152+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:40.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:40.671+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:40.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:40.671+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:40.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:40.671+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:40.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:40.672+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:40.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:40.719+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:40.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:40.719+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:40.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:40.719+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:40.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:40.720+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:41.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:41.193+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:41.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:41.193+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:41.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:41.193+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:41.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:41.194+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:41.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:41.744+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:41.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:41.744+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:41.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:41.744+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:41.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:41.744+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:41.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:41.640+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:41.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:41.640+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:41.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:41.640+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:41.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:41.641+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:42.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:42.237+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:42.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:42.237+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:42.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:42.237+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:42.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:42.237+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:42.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:42.685+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:42.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:42.685+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:42.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:42.685+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:42.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:42.686+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:42.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:42.713+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:42.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:42.713+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:42.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:42.713+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:42.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:42.713+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:43.432 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:15:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:15:43.067+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 22845 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:15:43.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:43.228+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:43.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:43.228+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:43.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:43.228+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:43.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:43.229+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:43.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:43.702+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:43.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:43.702+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:43.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:43.702+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:43.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:43.702+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:43.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:43.728+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:43.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:43.728+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:43.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:43.728+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:43.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:43.729+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:44.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:44.251+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:44.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:44.251+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:44.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:44.251+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:44.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:44.251+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:45.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:44.742+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:45.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:44.742+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:45.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:44.742+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:45.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:44.742+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:45.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:44.742+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:45.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:44.742+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:45.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:44.742+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:45.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:44.742+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:45.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:45.204+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:45.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:45.204+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:45.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:45.204+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:45.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:45.205+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:46.157 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:45.715+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:46.157 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:45.715+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:46.157 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:45.715+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:46.158 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:45.715+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:46.158 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:45.703+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:46.158 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:45.703+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:46.158 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:45.703+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:46.158 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:45.704+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:46.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:46.184+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:46.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:46.184+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:46.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:46.184+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:46.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:46.184+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:46.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:46.678+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:46.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:46.678+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:46.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:46.678+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:46.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:46.679+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:46.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:46.709+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:46.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:46.709+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:46.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:46.709+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:46.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:46.710+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:47.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:47.232+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:47.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:47.232+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:47.686 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:47.232+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:47.686 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:47.233+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:48.064 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:47.742+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:48.064 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:47.742+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:48.064 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:47.742+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:48.064 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:47.742+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:48.065 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:47.719+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:48.065 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:47.719+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:48.065 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:47.719+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:48.065 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:47.720+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:48.432 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:15:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:15:48.071+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 22935 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:15:48.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:48.244+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:48.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:48.244+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:48.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:48.244+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:48.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:48.244+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:49.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:48.764+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:49.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:48.764+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:49.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:48.764+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:49.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:48.765+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:49.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:48.737+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:49.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:48.737+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:49.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:48.737+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:49.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:48.738+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:49.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:49.229+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:49.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:49.229+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:49.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:49.229+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:49.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:49.230+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:50.180 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:49.717+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:50.180 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:49.717+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:50.181 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:49.717+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:50.181 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:49.718+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:50.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:49.751+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:50.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:49.751+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:50.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:49.751+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:50.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:49.751+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:50.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:50.228+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:50.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:50.228+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:50.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:50.228+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:50.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:50.229+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:51.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:50.794+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:51.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:50.794+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:51.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:50.794+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:51.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:50.794+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:51.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:50.692+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:51.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:50.692+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:51.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:50.692+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:51.175 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:50.693+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:51.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:51.181+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:51.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:51.181+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:51.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:51.181+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:51.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:51.181+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:51.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:51.646+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:51.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:51.646+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:51.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:51.646+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:51.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:51.646+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:51.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:51.783+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:51.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:51.783+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:51.935 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:51.783+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:51.935 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:51.784+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:52.301 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:52.230+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:52.301 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:52.230+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:52.302 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:52.230+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:52.302 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:52.230+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:52.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:52.628+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:52.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:52.628+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:52.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:52.628+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:52.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:52.629+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:52.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:52.786+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:52.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:52.786+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:52.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:52.786+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:52.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:52.786+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:53.432 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:15:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:15:53.075+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 23024 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:15:53.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:53.245+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:53.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:53.245+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:53.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:53.245+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:53.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:53.245+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:53.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:53.664+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:53.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:53.664+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:53.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:53.664+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:53.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:53.664+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:53.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:53.820+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:53.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:53.820+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:53.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:53.820+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:53.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:53.820+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:54.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:54.252+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:54.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:54.252+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:54.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:54.252+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:54.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:54.253+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:55.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:54.689+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:55.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:54.689+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:55.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:54.689+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:55.184 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:54.690+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:55.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:54.798+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:55.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:54.798+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:55.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:54.798+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:55.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:54.798+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:55.667 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:55.255+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:55.667 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:55.255+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:55.668 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:55.255+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:55.668 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:55.256+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:55.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:55.674+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:55.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:55.674+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:55.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:55.674+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:55.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:55.674+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:55.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:55.811+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:55.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:55.811+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:55.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:55.811+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:55.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:55.811+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:56.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:56.268+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:56.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:56.268+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:56.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:56.268+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:56.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:56.268+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:57.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:56.720+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:57.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:56.720+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:57.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:56.720+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:57.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:56.721+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:57.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:56.806+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:57.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:56.806+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:57.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:56.806+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:57.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:56.807+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:57.682 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:57.285+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:57.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:57.285+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:57.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:57.285+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:57.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:57.285+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:58.072 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:57.777+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:58.072 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:57.777+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:58.072 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:57.777+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:58.072 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:57.777+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:58.073 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:57.760+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:58.073 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:57.760+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:58.073 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:57.760+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:58.073 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:57.760+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:58.432 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:15:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:15:58.078+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 23116 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:15:58.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:58.272+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:58.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:58.272+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:58.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:58.272+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:58.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:58.272+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:59.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:58.763+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:59.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:58.763+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:59.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:58.763+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:15:59.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:58.764+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:15:59.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:58.799+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:59.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:58.799+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:59.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:58.799+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:15:59.184 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:58.799+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:15:59.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:59.246+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:59.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:59.246+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:59.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:59.246+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:15:59.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:15:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:15:59.246+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:00.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:59.766+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:00.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:59.766+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:00.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:59.766+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:00.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:15:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:15:59.766+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:00.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:59.801+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:00.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:59.801+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:00.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:59.801+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:00.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:15:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:15:59.801+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:00.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:00.242+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:00.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:00.242+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:00.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:00.242+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:00.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:00.243+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:01.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:00.793+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:01.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:00.793+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:01.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:00.793+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:01.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:00.793+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:01.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:00.779+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:01.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:00.779+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:01.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:00.779+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:01.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:00.779+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:01.517 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:01.203+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:01.518 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:01.203+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:01.518 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:01.203+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:01.518 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:01.203+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:01.815 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:01.763+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:01.815 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:01.763+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:01.815 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:01.763+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:01.815 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:01.763+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:02.158 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:01.821+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:02.158 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:01.821+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:02.158 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:01.821+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:02.159 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:01.822+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:02.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:02.165+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:02.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:02.165+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:02.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:02.165+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:02.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:02.165+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:03.076 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:02.868+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:03.076 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:02.868+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:03.076 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:02.868+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:03.076 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:02.868+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:03.076 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:02.753+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:03.077 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:02.753+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:03.077 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:02.753+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:03.077 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:02.753+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:03.433 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:16:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:16:03.082+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 23206 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:16:03.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:03.198+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:03.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:03.198+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:03.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:03.198+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:03.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:03.198+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:03.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:03.707+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:03.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:03.707+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:03.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:03.707+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:03.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:03.708+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:03.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:03.863+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:03.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:03.863+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:03.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:03.863+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:03.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:03.863+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:04.589 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:04.217+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:04.590 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:04.217+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:04.590 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:04.217+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:04.590 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:04.217+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:04.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:04.672+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:04.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:04.672+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:04.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:04.672+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:04.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:04.672+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:04.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:04.841+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:04.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:04.841+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:04.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:04.841+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:04.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:04.841+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:05.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:05.174+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:05.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:05.174+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:05.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:05.174+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:05.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:05.174+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:06.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:05.703+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:06.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:05.703+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:06.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:05.703+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:06.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:05.703+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:06.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:05.810+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:06.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:05.810+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:06.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:05.810+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:06.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:05.810+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:06.669 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:06.208+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:06.669 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:06.208+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:06.669 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:06.208+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:06.669 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:06.208+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:06.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:06.675+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:06.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:06.675+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:06.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:06.675+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:06.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:06.676+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:06.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:06.765+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:06.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:06.765+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:06.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:06.765+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:06.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:06.766+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:07.645 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:07.255+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:07.645 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:07.255+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:07.645 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:07.255+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:07.645 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:07.255+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:07.932 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:07.774+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:07.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:07.774+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:07.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:07.774+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:07.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:07.774+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:07.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:07.652+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:07.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:07.652+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:07.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:07.652+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:07.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:07.652+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:08.433 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:16:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:16:08.086+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 23298 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:16:08.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:08.235+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:08.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:08.235+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:08.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:08.235+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:08.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:08.235+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:08.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:08.724+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:08.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:08.724+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:08.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:08.724+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:08.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:08.725+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:08.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:08.628+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:08.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:08.628+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:08.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:08.628+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:08.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:08.628+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:09.594 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:09.197+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:09.594 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:09.197+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:09.594 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:09.197+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:09.594 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:09.197+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:09.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:09.600+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:09.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:09.600+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:09.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:09.600+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:09.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:09.601+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:09.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:09.734+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:09.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:09.734+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:09.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:09.734+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:09.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:09.734+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:10.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:10.245+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:10.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:10.245+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:10.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:10.245+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:10.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:10.245+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:10.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:10.557+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:10.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:10.557+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:10.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:10.557+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:10.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:10.557+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:10.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:10.783+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:10.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:10.783+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:10.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:10.783+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:10.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:10.783+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:11.551 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:11.225+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:11.552 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:11.225+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:11.552 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:11.225+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:11.552 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:11.225+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:11.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:11.558+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:11.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:11.558+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:11.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:11.558+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:11.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:11.558+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:11.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:11.790+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:11.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:11.790+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:11.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:11.790+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:11.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:11.790+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:12.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:12.227+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:12.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:12.227+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:12.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:12.227+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:12.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:12.228+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:12.816 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:12.558+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:12.817 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:12.558+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:12.817 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:12.558+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:12.817 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:12.558+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:13.083 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:12.823+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:13.084 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:12.823+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:13.084 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:12.823+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:13.084 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:12.823+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:13.433 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:16:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:16:13.090+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 23386 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:16:13.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:13.265+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:13.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:13.265+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:13.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:13.265+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:13.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:13.266+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:13.805 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:13.511+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:13.806 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:13.511+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:13.806 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:13.511+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:13.806 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:13.512+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:14.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:13.868+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:14.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:13.868+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:14.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:13.868+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:14.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:13.868+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:14.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:14.492+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:14.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:14.492+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:14.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:14.492+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:14.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:14.492+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:14.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:14.247+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:14.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:14.247+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:14.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:14.247+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:14.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:14.247+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:15.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:14.830+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:15.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:14.830+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:15.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:14.830+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:15.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:14.830+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:15.448 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:15.204+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:15.448 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:15.204+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:15.448 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:15.204+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:15.449 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:15.204+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:15.753 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:15.517+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:15.754 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:15.517+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:15.754 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:15.517+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:15.754 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:15.517+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:16.145 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:15.780+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:16.145 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:15.780+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:16.146 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:15.780+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:16.146 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:15.780+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:16.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:16.219+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:16.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:16.219+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:16.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:16.219+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:16.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:16.220+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:16.757 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:16.468+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:16.757 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:16.468+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:16.757 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:16.468+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:16.757 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:16.468+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:17.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:16.764+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:17.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:16.764+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:17.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:16.764+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:17.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:16.764+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:17.491 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:17.204+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:17.491 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:17.204+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:17.491 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:17.204+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:17.491 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:17.204+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:17.755 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:17.497+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:17.755 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:17.497+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:17.755 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:17.497+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:17.755 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:17.498+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:18.087 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:17.761+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:18.087 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:17.761+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:18.087 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:17.761+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:18.088 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:17.761+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:18.433 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:16:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:16:18.093+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 23478 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:16:18.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:18.246+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:18.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:18.246+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:18.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:18.246+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:18.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:18.246+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:18.753 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:18.488+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:18.753 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:18.488+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:18.753 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:18.488+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:18.753 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:18.489+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:19.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:18.760+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:19.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:18.760+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:19.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:18.760+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:19.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:18.760+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:19.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:19.475+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:19.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:19.475+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:19.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:19.475+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:19.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:19.475+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:19.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:19.246+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:19.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:19.246+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:19.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:19.246+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:19.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:19.246+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:20.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:19.758+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:20.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:19.758+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:20.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:19.758+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:20.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:19.758+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:20.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:20.260+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:20.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:20.260+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:20.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:20.260+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:20.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:20.260+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:20.773 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:20.507+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:20.773 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:20.507+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:20.773 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:20.507+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:20.773 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:20.507+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:21.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:20.780+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:21.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:20.780+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:21.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:20.780+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:21.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:20.780+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:21.500 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:21.257+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:21.500 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:21.257+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:21.500 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:21.257+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:21.501 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:21.257+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:21.501 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:21.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:21.501 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:21.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:21.501 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:21.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:21.501 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:21.500+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:22.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:21.798+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:22.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:21.798+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:22.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:21.798+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:22.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:21.799+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:22.460 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:22.293+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:22.460 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:22.293+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:22.460 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:22.293+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:22.460 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:22.293+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:22.771 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:22.467+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:22.771 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:22.467+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:22.771 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:22.467+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:22.771 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:22.467+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:23.022 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:22.777+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:23.022 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:22.777+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:23.022 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:22.777+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:23.022 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:22.777+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:23.274 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:16:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:16:23.097+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 23568 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:16:23.615 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:23.281+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:23.615 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:23.281+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:23.615 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:23.281+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:23.615 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:23.281+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:23.616 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:23.422+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:23.616 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:23.422+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:23.616 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:23.422+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:23.616 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:23.422+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:23.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:23.756+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:23.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:23.756+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:23.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:23.756+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:23.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:23.756+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:24.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:24.381+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:24.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:24.381+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:24.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:24.381+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:24.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:24.381+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:24.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:24.282+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:24.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:24.282+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:24.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:24.282+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:24.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:24.282+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:25.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:24.779+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:25.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:24.779+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:25.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:24.779+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:25.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:24.779+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:25.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:25.381+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:25.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:25.381+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:25.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:25.381+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:25.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:25.381+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:25.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:25.241+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:25.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:25.241+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:25.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:25.241+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:25.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:25.241+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:26.093 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:25.758+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:26.094 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:25.758+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:26.094 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:25.758+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:26.094 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:25.758+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:26.409 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:26.288+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:26.410 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:26.288+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:26.410 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:26.288+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:26.410 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:26.288+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:26.666 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:26.415+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:26.667 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:26.415+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:26.667 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:26.415+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:26.667 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:26.415+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:26.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:26.715+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:26.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:26.715+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:26.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:26.715+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:26.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:26.716+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:27.241 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:27.245+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:27.242 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:27.245+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:27.242 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:27.245+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:27.242 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:27.245+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:27.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:27.449+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:27.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:27.449+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:27.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:27.449+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:27.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:27.449+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:28.095 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:27.740+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:28.095 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:27.740+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:28.095 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:27.740+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:28.096 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:27.740+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:28.405 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:16:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:16:28.101+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 23659 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:16:28.405 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:28.221+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:28.406 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:28.221+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:28.406 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:28.221+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:28.406 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:28.221+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:28.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:28.411+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:28.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:28.411+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:28.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:28.411+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:28.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:28.412+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:29.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:28.786+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:29.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:28.786+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:29.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:28.786+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:29.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:28.787+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:29.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:29.417+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:29.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:29.417+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:29.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:29.417+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:29.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:29.418+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:29.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:29.221+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:29.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:29.221+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:29.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:29.221+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:29.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:29.222+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:30.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:29.770+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:30.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:29.770+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:30.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:29.770+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:30.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:29.770+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:30.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:30.257+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:30.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:30.257+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:30.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:30.257+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:30.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:30.257+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:30.434 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:30.426+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:30.434 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:30.426+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:30.434 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:30.426+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:30.434 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:30.427+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:31.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:30.791+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:31.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:30.791+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:31.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:30.791+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:31.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:30.791+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:31.584 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:31.472+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:31.585 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:31.472+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:31.585 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:31.472+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:31.585 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:31.472+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:31.586 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:31.222+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:31.586 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:31.222+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:31.586 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:31.222+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:31.587 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:31.222+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:31.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:31.744+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:31.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:31.744+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:31.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:31.744+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:31.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:31.745+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:32.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:32.234+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:32.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:32.234+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:32.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:32.234+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:32.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:32.234+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:32.771 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:32.481+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:32.771 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:32.481+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:32.771 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:32.481+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:32.771 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:32.482+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:33.100 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:32.777+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:33.101 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:32.777+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:33.101 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:32.777+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:33.101 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:32.777+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:33.428 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:16:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:16:33.105+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 23758 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:16:33.429 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:33.263+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:33.429 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:33.263+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:33.429 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:33.263+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:33.429 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:33.263+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:33.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:33.434+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:33.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:33.435+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:33.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:33.435+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:33.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:33.435+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:34.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:33.739+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:34.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:33.739+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:34.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:33.739+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:34.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:33.740+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:34.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:34.311+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:34.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:34.311+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:34.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:34.311+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:34.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:34.311+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:34.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:34.389+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:34.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:34.389+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:34.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:34.389+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:34.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:34.390+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:35.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:34.729+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:35.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:34.729+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:35.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:34.729+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:35.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:34.729+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:35.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:35.345+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:35.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:35.345+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:35.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:35.345+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:35.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:35.345+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:35.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:35.418+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:35.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:35.418+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:35.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:35.418+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:35.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:35.419+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:36.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:35.697+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:36.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:35.697+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:36.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:35.697+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:36.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:35.697+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:36.627 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:36.300+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:36.627 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:36.300+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:36.627 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:36.300+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:36.627 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:36.300+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:36.628 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:36.451+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:36.628 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:36.451+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:36.628 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:36.451+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:36.628 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:36.451+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:36.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:36.691+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:36.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:36.691+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:36.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:36.691+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:36.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:36.691+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:37.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:37.449+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:37.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:37.449+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:37.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:37.449+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:37.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:37.450+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:37.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:37.345+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:37.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:37.345+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:37.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:37.345+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:37.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:37.345+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:38.103 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:37.718+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:38.103 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:37.718+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:38.103 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:37.718+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:38.103 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:37.719+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:38.382 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:16:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:16:38.109+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 23850 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:16:38.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:38.439+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:38.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:38.439+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:38.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:38.439+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:38.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:38.439+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:38.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:38.388+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:38.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:38.388+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:38.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:38.388+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:38.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:38.388+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:39.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:38.736+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:39.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:38.736+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:39.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:38.736+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:39.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:38.737+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:39.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:39.483+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:39.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:39.483+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:39.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:39.483+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:39.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:39.484+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:39.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:39.370+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:39.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:39.370+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:39.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:39.370+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:39.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:39.371+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:40.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:39.731+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:40.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:39.731+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:40.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:39.731+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:40.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:39.732+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:40.525 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:40.413+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:40.525 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:40.413+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:40.525 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:40.413+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:40.526 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:40.413+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:40.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:40.702+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:40.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:40.702+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:40.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:40.702+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:40.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:40.702+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:40.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:40.531+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:40.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:40.531+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:40.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:40.531+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:40.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:40.531+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:41.728 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:41.545+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:41.729 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:41.545+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:41.729 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:41.545+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:41.729 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:41.545+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:41.729 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:41.444+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:41.730 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:41.444+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:41.730 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:41.444+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:41.730 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:41.444+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:42.009 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:41.735+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:42.009 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:41.735+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:42.009 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:41.735+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:42.009 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:41.735+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:42.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:42.538+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:42.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:42.538+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:42.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:42.538+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:42.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:42.538+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:42.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:42.435+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:42.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:42.435+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:42.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:42.435+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:42.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:42.435+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:43.107 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:42.722+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:43.107 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:42.722+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:43.107 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:42.722+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:43.107 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:42.722+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:43.394 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:16:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:16:43.113+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 23940 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:16:43.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:43.562+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:43.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:43.562+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:43.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:43.562+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:43.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:43.563+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:43.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:43.400+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:43.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:43.400+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:43.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:43.400+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:43.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:43.400+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:44.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:43.695+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:44.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:43.695+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:44.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:43.695+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:44.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:43.696+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:44.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:44.389+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:44.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:44.389+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:44.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:44.389+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:44.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:44.389+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:44.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:44.542+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:44.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:44.542+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:44.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:44.542+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:44.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:44.543+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:45.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:44.723+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:45.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:44.723+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:45.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:44.723+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:45.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:44.723+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:45.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:45.388+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:45.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:45.388+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:45.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:45.388+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:45.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:45.388+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:45.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:45.563+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:45.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:45.563+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:45.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:45.563+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:45.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:45.563+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:46.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:45.753+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:46.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:45.753+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:46.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:45.753+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:46.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:45.754+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:46.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:46.607+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:46.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:46.607+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:46.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:46.607+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:46.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:46.607+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:46.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:46.408+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:46.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:46.408+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:46.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:46.408+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:46.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:46.409+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:47.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:46.797+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:47.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:46.797+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:47.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:46.797+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:47.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:46.797+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:47.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:47.369+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:47.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:47.369+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:47.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:47.369+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:47.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:47.369+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:47.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:47.574+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:47.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:47.574+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:47.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:47.574+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:47.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:47.575+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:48.111 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:47.810+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:48.111 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:47.810+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:48.111 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:47.810+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:48.111 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:47.810+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:48.433 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:16:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:16:48.117+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 24032 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:16:48.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:48.342+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:48.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:48.342+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:48.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:48.342+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:48.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:48.342+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:48.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:48.821+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:48.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:48.821+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:48.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:48.821+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:48.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:48.821+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:48.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:48.604+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:48.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:48.604+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:48.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:48.604+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:48.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:48.604+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:49.572 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:49.317+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:49.572 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:49.317+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:49.572 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:49.317+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:49.572 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:49.317+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:49.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:49.578+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:49.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:49.578+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:49.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:49.578+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:49.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:49.578+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:49.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:49.805+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:49.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:49.805+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:49.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:49.805+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:49.935 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:49.805+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:50.552 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:50.288+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:50.553 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:50.288+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:50.553 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:50.288+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:50.553 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:50.288+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:50.835 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:50.559+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:50.835 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:50.559+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:50.835 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:50.559+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:50.835 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:50.559+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:51.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:50.840+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:51.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:50.840+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:51.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:50.840+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:51.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:50.841+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:51.597 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:51.304+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:51.597 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:51.304+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:51.597 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:51.304+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:51.598 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:51.304+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:51.880 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:51.602+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:51.880 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:51.602+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:51.880 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:51.602+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:51.881 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:51.603+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:52.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:51.886+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:52.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:51.886+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:52.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:51.886+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:52.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:51.886+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:52.622 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:52.332+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:52.622 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:52.332+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:52.623 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:52.332+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:52.623 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:52.332+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:52.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:52.846+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:52.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:52.846+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:52.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:52.846+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:52.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:52.846+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:52.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:52.628+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:52.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:52.628+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:52.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:52.628+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:52.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:52.628+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:53.376 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:16:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:16:53.121+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 24121 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:16:53.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:53.611+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:53.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:53.611+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:53.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:53.611+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:53.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:53.611+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:53.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:53.382+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:53.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:53.382+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:53.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:53.382+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:53.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:53.382+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:54.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:53.829+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:54.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:53.829+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:54.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:53.829+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:54.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:53.829+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:54.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:54.562+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:54.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:54.562+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:54.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:54.562+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:54.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:54.562+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:54.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:54.396+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:54.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:54.396+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:54.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:54.396+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:54.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:54.396+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:55.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:54.828+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:55.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:54.828+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:55.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:54.828+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:55.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:54.829+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:55.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:55.604+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:55.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:55.604+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:55.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:55.604+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:55.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:55.605+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:55.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:55.375+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:55.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:55.375+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:55.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:55.375+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:55.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:55.375+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:56.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:55.855+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:56.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:55.855+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:56.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:55.855+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:56.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:55.856+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:56.634 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:56.381+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:56.634 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:56.382+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:56.634 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:56.382+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:56.634 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:56.382+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:56.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:56.867+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:56.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:56.867+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:56.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:56.867+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:56.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:56.867+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:56.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:56.640+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:56.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:56.640+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:56.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:56.640+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:56.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:56.640+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:57.605 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:57.337+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:57.605 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:57.337+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:57.605 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:57.337+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:57.605 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:57.337+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:57.892 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:57.610+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:57.892 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:57.610+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:57.892 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:57.610+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:57.893 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:57.611+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:58.143 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:16:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:16:58.125+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 24213 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:16:58.144 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:57.898+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:58.144 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:57.898+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:58.144 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:57.898+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:58.144 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:57.898+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:58.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:58.332+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:58.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:58.332+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:58.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:58.332+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:58.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:58.332+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:58.880 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:58.608+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:58.880 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:58.608+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:58.881 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:58.608+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:58.881 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:58.608+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:59.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:58.886+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:59.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:58.886+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:59.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:58.886+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:59.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:58.886+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:59.629 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:59.365+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:59.629 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:59.365+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:59.629 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:59.365+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:16:59.629 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:16:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:16:59.365+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:16:59.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:59.859+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:59.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:59.859+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:59.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:59.859+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:16:59.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:16:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:16:59.859+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:16:59.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:59.634+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:59.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:59.634+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:59.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:59.634+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:16:59.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:16:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:16:59.635+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:00.594 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:00.352+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:00.594 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:00.352+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:00.594 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:00.352+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:00.595 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:00.353+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:00.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:00.843+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:00.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:00.843+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:00.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:00.843+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:00.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:00.843+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:00.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:00.600+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:00.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:00.600+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:00.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:00.600+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:00.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:00.600+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:01.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:01.371+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:01.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:01.371+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:01.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:01.371+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:01.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:01.372+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:01.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:01.566+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:01.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:01.566+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:01.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:01.566+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:01.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:01.566+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:02.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:01.857+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:02.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:01.857+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:02.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:01.857+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:02.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:01.857+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:02.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:02.386+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:02.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:02.386+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:02.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:02.386+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:02.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:02.386+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:02.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:02.586+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:02.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:02.586+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:02.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:02.586+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:02.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:02.586+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:03.123 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:02.817+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:03.123 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:02.817+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:03.124 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:02.817+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:03.124 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:02.817+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:03.433 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:17:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:17:03.129+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 24303 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:17:03.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:03.365+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:03.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:03.365+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:03.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:03.365+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:03.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:03.365+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:03.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:03.622+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:03.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:03.622+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:03.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:03.622+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:03.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:03.623+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:03.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:03.772+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:03.935 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:03.772+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:03.935 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:03.772+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:03.935 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:03.772+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:04.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:04.609+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:04.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:04.609+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:04.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:04.609+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:04.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:04.610+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:04.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:04.413+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:04.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:04.413+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:04.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:04.413+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:04.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:04.413+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:05.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:04.784+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:05.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:04.784+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:05.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:04.784+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:05.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:04.784+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:05.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:05.608+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:05.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:05.608+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:05.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:05.608+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:05.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:05.608+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:05.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:05.371+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:05.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:05.371+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:05.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:05.371+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:05.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:05.372+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:06.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:05.814+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:06.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:05.814+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:06.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:05.814+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:06.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:05.814+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:06.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:06.359+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:06.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:06.359+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:06.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:06.359+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:06.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:06.359+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:06.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:06.577+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:06.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:06.577+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:06.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:06.577+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:06.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:06.577+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:07.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:06.790+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:07.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:06.790+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:07.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:06.790+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:07.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:06.790+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:07.683 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:07.357+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:07.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:07.357+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:07.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:07.357+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:07.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:07.358+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:07.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:07.595+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:07.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:07.595+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:07.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:07.595+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:07.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:07.595+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:08.127 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:07.802+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:08.128 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:07.802+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:08.128 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:07.802+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:08.128 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:07.802+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:08.434 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:17:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:17:08.133+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 24395 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:17:08.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:08.345+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:08.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:08.345+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:08.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:08.345+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:08.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:08.345+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:08.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:08.800+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:08.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:08.800+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:08.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:08.800+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:08.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:08.800+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:08.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:08.570+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:08.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:08.570+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:08.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:08.570+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:08.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:08.570+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:09.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:09.524+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:09.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:09.524+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:09.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:09.524+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:09.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:09.524+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:09.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:09.323+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:09.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:09.323+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:09.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:09.323+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:09.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:09.323+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:10.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:09.776+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:10.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:09.776+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:10.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:09.776+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:10.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:09.776+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:10.542 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:10.324+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:10.542 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:10.324+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:10.542 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:10.324+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:10.542 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:10.325+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:10.795 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:10.547+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:10.796 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:10.547+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:10.796 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:10.547+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:10.796 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:10.548+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:11.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:10.801+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:11.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:10.801+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:11.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:10.801+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:11.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:10.801+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:11.566 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:11.317+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:11.566 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:11.317+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:11.566 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:11.317+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:11.567 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:11.317+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:11.828 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:11.572+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:11.829 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:11.572+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:11.829 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:11.572+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:11.829 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:11.572+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:12.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:11.835+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:12.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:11.835+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:12.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:11.835+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:12.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:11.835+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:12.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:12.320+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:12.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:12.320+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:12.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:12.320+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:12.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:12.320+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:12.933 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:12.799+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:12.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:12.799+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:12.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:12.799+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:12.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:12.799+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:12.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:12.610+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:12.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:12.610+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:12.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:12.610+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:12.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:12.610+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:13.433 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:17:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:17:13.137+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 24484 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:17:13.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:13.329+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:13.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:13.329+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:13.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:13.329+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:13.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:13.329+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:13.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:13.658+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:13.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:13.658+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:13.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:13.658+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:13.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:13.659+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:13.935 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:13.756+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:13.935 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:13.756+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:13.935 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:13.756+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:13.935 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:13.757+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:14.606 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:14.338+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:14.606 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:14.338+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:14.606 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:14.338+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:14.606 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:14.338+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:14.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:14.760+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:14.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:14.760+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:14.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:14.760+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:14.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:14.761+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:14.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:14.610+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:14.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:14.610+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:14.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:14.610+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:14.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:14.611+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:15.571 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:15.358+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:15.571 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:15.358+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:15.571 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:15.358+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:15.571 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:15.359+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:15.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:15.576+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:15.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:15.576+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:15.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:15.576+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:15.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:15.576+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:15.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:15.764+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:15.935 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:15.764+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:15.935 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:15.764+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:15.935 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:15.764+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:16.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:16.545+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:16.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:16.545+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:16.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:16.545+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:16.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:16.545+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:16.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:16.326+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:16.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:16.326+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:16.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:16.326+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:16.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:16.326+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:17.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:16.739+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:17.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:16.739+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:17.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:16.739+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:17.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:16.739+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:17.562 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:17.281+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:17.563 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:17.281+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:17.563 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:17.281+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:17.563 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:17.282+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:17.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:17.732+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:17.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:17.732+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:17.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:17.732+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:17.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:17.733+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:17.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:17.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:17.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:17.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:17.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:17.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:17.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:17.568+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:18.433 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:17:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:17:18.141+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 24576 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:17:18.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:18.310+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:18.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:18.310+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:18.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:18.311+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:18.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:18.311+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:18.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:18.740+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:18.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:18.740+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:18.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:18.740+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:18.935 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:18.740+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:18.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:18.575+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:18.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:18.575+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:18.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:18.575+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:18.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:18.575+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:19.551 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:19.263+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:19.551 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:19.263+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:19.551 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:19.263+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:19.551 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:19.263+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:19.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:19.556+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:19.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:19.556+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:19.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:19.556+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:19.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:19.557+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:19.935 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:19.691+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:19.936 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:19.691+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:19.936 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:19.691+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:19.936 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:19.691+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:20.575 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:20.306+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:20.575 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:20.306+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:20.575 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:20.306+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:20.575 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:20.306+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:20.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:20.730+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:20.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:20.730+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:20.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:20.730+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:20.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:20.730+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:20.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:20.580+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:20.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:20.580+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:20.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:20.580+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:20.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:20.580+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:21.591 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:21.346+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:21.591 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:21.346+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:21.591 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:21.346+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:21.591 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:21.347+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:21.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:21.703+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:21.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:21.703+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:21.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:21.703+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:21.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:21.703+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:21.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:21.596+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:21.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:21.596+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:21.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:21.596+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:21.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:21.597+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:22.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:22.318+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:22.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:22.318+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:22.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:22.318+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:22.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:22.318+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:22.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:22.551+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:22.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:22.551+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:22.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:22.551+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:22.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:22.552+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:23.140 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:22.732+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:23.140 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:22.732+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:23.140 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:22.732+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:23.140 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:22.732+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:23.434 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:17:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:17:23.145+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 24666 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:17:23.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:23.354+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:23.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:23.354+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:23.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:23.354+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:23.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:23.355+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:23.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:23.527+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:23.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:23.527+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:23.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:23.527+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:23.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:23.527+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:23.937 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:23.751+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:23.937 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:23.751+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:23.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:23.751+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:23.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:23.751+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:24.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:24.320+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:24.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:24.320+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:24.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:24.320+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:24.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:24.320+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:24.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:24.492+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:24.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:24.492+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:24.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:24.492+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:24.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:24.493+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:25.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:24.768+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:25.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:24.768+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:25.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:24.768+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:25.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:24.768+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:25.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:25.475+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:25.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:25.475+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:25.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:25.475+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:25.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:25.475+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:25.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:25.305+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:25.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:25.305+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:25.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:25.305+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:25.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:25.305+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:26.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:25.797+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:26.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:25.797+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:26.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:25.797+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:26.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:25.798+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:26.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:26.451+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:26.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:26.451+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:26.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:26.451+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:26.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:26.451+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:26.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:26.322+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:26.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:26.322+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:26.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:26.322+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:26.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:26.322+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:27.055 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:26.768+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:27.055 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:26.768+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:27.055 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:26.768+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:27.055 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:26.769+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:27.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:27.428+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:27.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:27.428+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:27.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:27.428+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:27.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:27.428+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:27.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:27.317+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:27.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:27.317+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:27.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:27.317+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:27.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:27.317+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:28.145 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:27.780+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:28.145 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:27.780+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:28.145 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:27.780+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:28.146 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:27.781+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:28.400 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:17:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:17:28.149+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 24757 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:17:28.401 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:28.314+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:28.401 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:28.314+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:28.401 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:28.314+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:28.401 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:28.314+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:28.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:28.406+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:28.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:28.406+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:28.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:28.406+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:28.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:28.406+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:29.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:28.748+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:29.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:28.748+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:29.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:28.748+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:29.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:28.748+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:29.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:29.444+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:29.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:29.444+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:29.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:29.444+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:29.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:29.444+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:29.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:29.275+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:29.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:29.275+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:29.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:29.275+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:29.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:29.276+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:30.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:29.771+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:30.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:29.771+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:30.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:29.771+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:30.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:29.771+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:30.468 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:30.281+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:30.469 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:30.281+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:30.469 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:30.281+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:30.469 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:30.281+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:30.809 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:30.474+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:30.809 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:30.474+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:30.809 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:30.474+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:30.809 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:30.474+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:31.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:30.814+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:31.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:30.814+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:31.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:30.814+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:31.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:30.815+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:31.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:31.445+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:31.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:31.445+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:31.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:31.445+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:31.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:31.445+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:31.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:31.263+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:31.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:31.263+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:31.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:31.263+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:31.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:31.263+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:32.097 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:31.791+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:32.097 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:31.791+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:32.097 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:31.791+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:32.097 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:31.791+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:32.395 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:32.248+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:32.395 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:32.248+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:32.396 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:32.248+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:32.396 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:32.248+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:32.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:32.400+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:32.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:32.400+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:32.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:32.400+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:32.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:32.401+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:33.148 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:32.775+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:33.148 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:32.775+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:33.149 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:32.775+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:33.149 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:32.776+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:33.402 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:17:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:17:33.153+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 24846 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:17:33.403 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:33.207+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:33.403 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:33.207+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:33.403 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:33.207+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:33.403 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:33.207+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:33.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:33.408+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:33.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:33.408+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:33.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:33.408+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:33.679 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:33.408+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:33.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:33.812+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:33.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:33.812+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:33.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:33.812+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:33.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:33.813+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:34.434 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:34.392+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:34.434 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:34.392+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:34.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:34.392+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:34.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:34.392+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:34.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:34.179+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:34.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:34.179+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:34.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:34.179+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:34.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:34.179+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:35.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:34.858+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:35.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:34.858+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:35.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:34.858+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:35.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:34.858+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:35.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:35.369+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:35.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:35.369+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:35.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:35.369+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:35.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:35.369+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:35.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:35.213+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:35.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:35.213+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:35.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:35.213+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:35.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:35.214+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:36.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:35.813+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:36.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:35.813+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:36.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:35.813+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:36.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:35.813+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:36.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:36.373+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:36.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:36.373+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:36.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:36.373+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:36.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:36.373+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:36.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:36.222+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:36.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:36.222+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:36.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:36.222+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:36.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:36.222+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:37.140 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:36.789+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:37.140 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:36.789+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:37.140 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:36.789+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:37.140 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:36.790+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:37.434 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:37.334+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:37.434 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:37.334+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:37.434 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:37.334+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:37.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:37.334+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:37.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:37.263+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:37.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:37.263+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:37.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:37.263+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:37.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:37.263+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:38.152 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:37.754+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:38.153 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:37.754+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:38.153 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:37.754+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:38.153 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:37.755+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:38.434 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:17:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:17:38.158+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 24938 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:17:38.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:38.250+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:38.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:38.250+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:38.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:38.250+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:38.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:38.250+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:38.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:38.343+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:38.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:38.343+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:38.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:38.343+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:38.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:38.343+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:39.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:38.792+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:39.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:38.792+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:39.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:38.792+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:39.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:38.793+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:39.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:39.376+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:39.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:39.376+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:39.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:39.376+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:39.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:39.376+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:39.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:39.290+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:39.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:39.290+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:39.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:39.290+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:39.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:39.290+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:40.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:39.805+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:40.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:39.805+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:40.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:39.805+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:40.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:39.805+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:40.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:40.404+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:40.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:40.405+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:40.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:40.405+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:40.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:40.405+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:40.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:40.258+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:40.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:40.258+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:40.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:40.258+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:40.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:40.258+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:41.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:40.844+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:41.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:40.844+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:41.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:40.844+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:41.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:40.844+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:41.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:41.363+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:41.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:41.363+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:41.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:41.363+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:41.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:41.363+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:41.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:41.246+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:41.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:41.246+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:41.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:41.246+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:41.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:41.246+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:42.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:41.884+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:42.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:41.884+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:42.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:41.884+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:42.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:41.884+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:42.434 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:42.398+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:42.434 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:42.398+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:42.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:42.398+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:42.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:42.398+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:42.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:42.233+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:42.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:42.233+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:42.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:42.233+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:42.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:42.234+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:43.157 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:42.876+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:43.157 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:42.876+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:43.157 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:42.876+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:43.157 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:42.876+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:43.416 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:17:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:17:43.162+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 25028 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:17:43.416 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:43.262+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:43.416 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:43.262+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:43.416 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:43.262+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:43.416 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:43.262+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:43.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:43.421+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:43.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:43.421+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:43.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:43.421+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:43.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:43.421+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:44.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:43.854+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:44.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:43.854+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:44.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:43.854+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:44.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:43.855+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:44.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:44.410+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:44.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:44.410+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:44.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:44.410+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:44.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:44.410+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:44.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:44.226+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:44.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:44.226+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:44.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:44.226+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:44.686 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:44.226+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:45.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:44.816+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:45.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:44.816+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:45.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:44.816+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:45.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:44.816+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:45.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:45.376+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:45.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:45.376+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:45.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:45.376+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:45.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:45.376+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:45.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:45.200+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:45.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:45.200+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:45.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:45.200+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:45.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:45.201+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:46.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:45.784+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:46.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:45.784+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:46.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:45.784+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:46.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:45.785+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:46.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:46.415+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:46.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:46.415+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:46.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:46.415+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:46.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:46.415+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:46.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:46.200+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:46.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:46.200+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:46.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:46.200+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:46.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:46.201+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:47.181 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:46.816+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:47.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:46.816+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:47.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:46.816+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:47.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:46.816+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:47.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:47.186+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:47.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:47.186+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:47.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:47.186+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:47.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:47.187+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:47.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:47.433+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:47.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:47.433+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:47.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:47.433+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:47.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:47.433+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:48.161 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:47.774+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:48.161 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:47.774+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:48.161 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:47.774+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:48.161 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:47.774+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:48.434 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:17:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:17:48.166+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 25120 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:17:48.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:48.178+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:48.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:48.178+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:48.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:48.178+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:48.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:48.178+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:48.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:48.392+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:48.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:48.392+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:48.436 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:48.392+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:48.436 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:48.392+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:49.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:48.771+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:49.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:48.771+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:49.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:48.771+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:49.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:48.771+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:49.684 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:49.201+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:49.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:49.201+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:49.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:49.201+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:49.685 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:49.202+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:49.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:49.405+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:49.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:49.405+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:49.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:49.405+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:49.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:49.405+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:50.176 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:49.758+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:50.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:49.758+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:50.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:49.758+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:50.177 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:49.758+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:50.434 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:50.396+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:50.434 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:50.396+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:50.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:50.396+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:50.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:50.396+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:50.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:50.182+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:50.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:50.182+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:50.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:50.182+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:50.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:50.182+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:51.132 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:50.781+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:51.136 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:50.781+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:51.136 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:50.781+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:51.137 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:50.781+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:51.392 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:51.135+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:51.392 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:51.135+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:51.392 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:51.135+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:51.392 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:51.135+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:51.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:51.397+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:51.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:51.397+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:51.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:51.397+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:51.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:51.397+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:52.088 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:51.782+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:52.089 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:51.782+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:52.089 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:51.782+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:52.089 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:51.782+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:52.419 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:52.093+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:52.419 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:52.093+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:52.419 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:52.093+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:52.419 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:52.094+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:52.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:52.423+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:52.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:52.423+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:52.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:52.423+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:52.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:52.424+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:53.040 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:52.769+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:53.040 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:52.769+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:53.040 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:52.769+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:53.040 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:52.769+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:53.371 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:17:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:17:53.170+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 25209 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:17:53.372 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:53.044+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:53.372 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:53.044+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:53.372 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:53.044+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:53.372 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:53.045+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:53.623 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:53.376+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:53.623 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:53.376+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:53.623 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:53.376+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:53.623 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:53.377+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:53.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:53.752+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:53.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:53.752+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:53.935 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:53.752+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:53.935 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:53.752+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:54.370 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:54.004+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:54.370 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:54.005+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:54.370 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:54.005+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:54.370 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:54.005+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:54.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:54.375+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:54.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:54.375+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:54.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:54.375+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:54.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:54.375+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:55.183 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:54.979+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:55.183 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:54.979+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:55.184 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:54.979+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:55.184 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:54.979+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:55.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:54.761+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:55.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:54.761+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:55.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:54.761+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:55.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:54.762+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:55.434 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:55.377+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:55.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:55.377+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:55.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:55.377+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:55.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:55.377+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:56.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:55.756+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:56.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:55.757+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:56.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:55.757+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:56.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:55.757+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:56.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:55.955+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:56.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:55.955+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:56.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:55.955+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:56.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:55.955+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:56.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:56.348+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:56.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:56.348+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:56.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:56.348+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:56.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:56.348+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:57.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:56.796+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:57.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:56.796+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:57.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:56.796+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:57.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:56.796+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:57.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:56.913+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:57.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:56.913+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:57.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:56.913+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:57.186 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:56.913+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:57.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:57.382+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:57.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:57.382+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:57.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:57.382+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:57.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:57.382+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:58.169 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:57.804+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:58.169 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:57.804+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:58.169 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:57.804+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:58.170 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:57.804+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:58.170 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:57.951+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:58.170 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:57.951+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:58.170 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:57.951+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:58.170 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:57.951+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:58.434 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:17:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:17:58.174+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 25301 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:17:58.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:58.392+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:58.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:58.392+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:58.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:58.392+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:58.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:58.392+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:59.184 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:58.946+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:59.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:58.946+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:59.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:58.946+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:17:59.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:58.946+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:17:59.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:58.815+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:59.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:58.815+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:59.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:58.815+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:17:59.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:58.815+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:17:59.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:59.365+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:59.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:59.365+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:59.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:59.365+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:17:59.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:17:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:17:59.365+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:00.184 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:59.958+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:00.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:59.958+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:00.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:59.958+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:00.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:17:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:17:59.958+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:00.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:59.778+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:00.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:59.778+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:00.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:59.778+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:00.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:17:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:17:59.778+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:00.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:00.381+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:00.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:00.381+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:00.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:00.381+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:00.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:00.381+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:01.184 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:00.963+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:01.184 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:00.963+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:01.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:00.963+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:01.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:00.963+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:01.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:00.756+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:01.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:00.756+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:01.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:00.756+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:01.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:00.757+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:01.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:01.336+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:01.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:01.336+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:01.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:01.336+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:01.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:01.336+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:02.184 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:01.970+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:02.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:01.970+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:02.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:01.970+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:02.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:01.970+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:02.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:01.792+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:02.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:01.792+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:02.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:01.792+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:02.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:01.793+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:02.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:02.329+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:02.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:02.329+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:02.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:02.329+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:02.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:02.329+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:03.173 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:02.972+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:03.174 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:02.972+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:03.174 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:02.972+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:03.174 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:02.972+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:03.174 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:02.820+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:03.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:02.820+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:03.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:02.820+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:03.175 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:02.820+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:03.434 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:18:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:18:03.178+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 25391 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:18:03.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:03.325+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:03.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:03.325+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:03.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:03.325+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:03.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:03.325+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:03.934 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:03.789+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:03.935 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:03.789+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:03.935 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:03.789+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:03.935 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:03.790+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:04.353 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:04.008+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:04.354 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:04.008+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:04.354 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:04.008+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:04.354 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:04.008+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:04.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:04.358+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:04.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:04.358+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:04.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:04.358+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:04.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:04.358+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:05.016 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:04.759+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:05.016 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:04.759+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:05.017 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:04.759+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:05.017 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:04.759+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:05.375 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:05.021+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:05.375 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:05.021+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:05.375 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:05.021+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:05.375 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:05.021+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:05.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:05.380+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:05.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:05.380+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:05.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:05.380+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:05.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:05.380+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:06.184 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:06.010+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:06.184 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:06.010+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:06.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:06.010+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:06.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:06.010+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:06.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:05.781+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:06.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:05.781+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:06.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:05.781+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:06.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:05.781+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:06.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:06.367+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:06.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:06.367+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:06.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:06.367+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:06.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:06.367+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:07.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:07.041+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:07.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:07.041+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:07.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:07.041+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:07.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:07.041+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:07.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:06.800+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:07.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:06.800+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:07.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:06.800+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:07.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:06.801+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:07.684 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:07.404+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:07.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:07.404+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:07.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:07.404+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:07.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:07.405+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:08.178 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:07.808+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:08.178 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:07.808+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:08.178 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:07.808+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:08.178 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:07.808+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:08.178 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:08.035+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:08.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:08.035+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:08.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:08.035+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:08.179 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:08.035+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:08.434 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:18:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:18:08.182+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 25483 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:18:08.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:08.418+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:08.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:08.418+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:08.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:08.418+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:08.435 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:08.419+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:09.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:09.002+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:09.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:09.002+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:09.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:09.002+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:09.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:09.002+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:09.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:08.810+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:09.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:08.810+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:09.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:08.810+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:09.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:08.811+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:09.856 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:09.460+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:09.857 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:09.460+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:09.857 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:09.460+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:09.857 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:09.460+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:10.184 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:10.025+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:10.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:10.025+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:10.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:10.025+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:10.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:10.025+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:10.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:09.861+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:10.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:09.861+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:10.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:09.861+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:10.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:09.862+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:10.898 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:10.492+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:10.898 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:10.492+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:10.898 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:10.492+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:10.898 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:10.493+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:11.184 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:11.067+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:11.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:11.067+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:11.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:11.067+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:11.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:11.067+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:11.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:10.903+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:11.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:10.903+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:11.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:10.903+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:11.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:10.903+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:11.918 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:11.461+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:11.918 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:11.461+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:11.918 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:11.461+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:11.918 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:11.462+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:12.184 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:12.077+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:12.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:12.077+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:12.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:12.077+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:12.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:12.078+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:12.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:11.922+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:12.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:11.922+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:12.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:11.922+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:12.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:11.923+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:12.882 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:12.483+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:12.882 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:12.483+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:12.882 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:12.483+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:12.882 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:12.484+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:13.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:12.886+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:13.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:12.886+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:13.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:12.886+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:13.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:12.887+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:13.183 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:13.111+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:13.183 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:13.111+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:13.183 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:13.111+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:13.183 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:13.111+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:13.434 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:18:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:18:13.187+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 25572 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:18:13.923 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:13.510+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:13.923 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:13.510+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:13.923 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:13.510+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:13.923 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:13.510+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:14.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:13.928+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:14.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:13.928+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:14.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:13.928+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:14.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:13.928+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:14.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:14.108+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:14.186 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:14.108+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:14.186 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:14.108+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:14.186 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:14.108+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:14.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:14.551+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:14.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:14.551+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:14.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:14.551+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:14.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:14.551+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:15.434 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:14.971+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:15.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:14.971+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:15.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:14.971+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:15.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:14.971+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:15.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:15.077+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:15.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:15.077+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:15.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:15.077+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:15.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:15.077+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:15.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:15.567+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:15.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:15.567+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:15.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:15.567+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:15.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:15.568+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:16.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:15.961+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:16.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:15.961+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:16.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:15.961+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:16.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:15.961+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:16.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:16.080+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:16.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:16.080+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:16.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:16.080+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:16.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:16.080+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:16.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:16.541+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:16.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:16.541+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:16.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:16.541+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:16.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:16.542+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:17.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:16.974+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:17.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:16.974+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:17.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:16.974+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:17.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:16.974+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:17.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:17.065+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:17.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:17.065+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:17.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:17.065+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:17.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:17.066+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:17.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:17.521+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:17.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:17.521+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:17.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:17.521+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:17.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:17.521+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:18.211 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:18:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:18:18.191+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 25664 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:18:18.211 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:17.961+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:18.211 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:17.961+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:18.211 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:17.961+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:18.212 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:17.962+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:18.212 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:18.044+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:18.212 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:18.044+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:18.212 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:18.044+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:18.212 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:18.045+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:18.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:18.472+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:18.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:18.472+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:18.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:18.472+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:18.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:18.472+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:19.434 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:18.988+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:19.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:18.988+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:19.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:18.988+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:19.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:18.988+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:19.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:19.011+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:19.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:19.011+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:19.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:19.011+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:19.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:19.011+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:19.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:19.460+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:19.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:19.460+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:19.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:19.460+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:19.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:19.460+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:20.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:20.012+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:20.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:20.012+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:20.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:20.012+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:20.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:20.012+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:20.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:20.026+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:20.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:20.026+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:20.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:20.026+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:20.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:20.026+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:20.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:20.485+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:20.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:20.485+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:20.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:20.485+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:20.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:20.486+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:21.434 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:21.022+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:21.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:21.022+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:21.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:21.022+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:21.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:21.022+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:21.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:21.044+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:21.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:21.044+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:21.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:21.044+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:21.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:21.044+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:21.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:21.522+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:21.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:21.522+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:21.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:21.522+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:21.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:21.522+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:22.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:22.038+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:22.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:22.038+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:22.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:22.038+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:22.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:22.038+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:22.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:22.036+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:22.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:22.036+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:22.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:22.036+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:22.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:22.036+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:22.832 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:22.488+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:22.832 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:22.488+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:22.832 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:22.488+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:22.832 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:22.488+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:23.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:22.994+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:23.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:22.994+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:23.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:22.994+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:23.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:22.994+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:23.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:23.047+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:23.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:23.047+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:23.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:23.047+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:23.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:23.047+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:23.477 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:18:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:18:23.195+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 25754 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:18:23.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:23.482+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:23.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:23.482+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:23.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:23.482+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:23.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:23.482+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:24.434 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:24.004+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:24.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:24.004+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:24.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:24.004+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:24.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:24.005+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:24.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:24.024+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:24.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:24.024+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:24.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:24.024+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:24.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:24.024+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:24.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:24.457+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:24.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:24.457+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:24.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:24.457+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:24.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:24.457+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:25.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:25.013+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:25.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:25.013+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:25.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:25.013+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:25.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:25.013+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:25.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:25.022+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:25.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:25.022+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:25.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:25.022+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:25.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:25.023+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:25.903 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:25.471+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:25.904 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:25.471+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:25.904 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:25.471+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:25.904 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:25.471+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:26.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:25.998+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:26.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:25.998+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:26.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:25.998+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:26.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:25.998+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:26.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:26.029+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:26.186 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:26.029+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:26.186 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:26.029+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:26.186 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:26.029+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:26.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:26.451+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:26.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:26.451+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:26.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:26.451+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:26.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:26.451+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:27.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:27.045+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:27.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:27.045+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:27.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:27.045+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:27.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:27.045+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:27.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:27.062+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:27.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:27.062+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:27.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:27.062+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:27.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:27.062+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:27.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:27.458+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:27.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:27.458+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:27.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:27.458+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:27.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:27.458+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:28.435 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:18:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:18:28.199+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 25846 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:18:28.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:28.086+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:28.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:28.086+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:28.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:28.086+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:28.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:28.087+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:28.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:28.083+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:28.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:28.083+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:28.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:28.083+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:28.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:28.083+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:28.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:28.508+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:28.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:28.508+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:28.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:28.508+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:28.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:28.508+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:29.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:29.077+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:29.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:29.077+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:29.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:29.077+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:29.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:29.077+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:29.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:29.131+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:29.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:29.131+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:29.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:29.131+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:29.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:29.131+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:29.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:29.474+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:29.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:29.474+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:29.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:29.474+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:29.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:29.475+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:30.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:30.117+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:30.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:30.117+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:30.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:30.117+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:30.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:30.117+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:30.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:30.095+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:30.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:30.095+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:30.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:30.095+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:30.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:30.095+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:30.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:30.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:30.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:30.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:30.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:30.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:30.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:30.500+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:31.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:31.105+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:31.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:31.105+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:31.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:31.105+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:31.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:31.105+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:31.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:31.117+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:31.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:31.117+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:31.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:31.117+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:31.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:31.117+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:31.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:31.525+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:31.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:31.525+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:31.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:31.525+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:31.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:31.525+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:32.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:32.116+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:32.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:32.116+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:32.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:32.116+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:32.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:32.117+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:32.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:32.086+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:32.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:32.086+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:32.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:32.086+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:32.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:32.086+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:32.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:32.505+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:32.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:32.505+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:32.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:32.505+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:32.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:32.505+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:33.326 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:18:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:18:33.203+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 25935 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:18:33.327 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:33.067+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:33.327 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:33.067+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:33.327 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:33.067+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:33.327 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:33.067+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:33.327 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:33.107+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:33.327 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:33.107+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:33.328 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:33.107+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:33.328 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:33.107+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:33.626 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:33.475+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:33.626 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:33.475+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:33.627 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:33.475+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:33.627 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:33.475+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:34.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:34.105+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:34.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:34.105+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:34.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:34.105+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:34.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:34.105+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:34.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:34.082+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:34.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:34.083+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:34.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:34.083+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:34.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:34.083+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:34.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:34.521+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:34.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:34.521+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:34.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:34.521+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:34.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:34.521+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:35.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:35.116+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:35.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:35.116+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:35.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:35.116+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:35.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:35.116+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:35.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:35.147+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:35.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:35.147+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:35.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:35.147+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:35.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:35.148+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:35.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:35.541+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:35.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:35.541+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:35.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:35.541+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:35.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:35.541+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:36.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:36.188+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:36.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:36.188+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:36.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:36.188+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:36.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:36.189+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:36.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:36.091+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:36.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:36.091+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:36.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:36.091+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:36.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:36.091+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:36.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:36.563+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:36.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:36.563+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:36.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:36.563+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:36.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:36.563+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:37.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:37.181+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:37.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:37.181+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:37.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:37.181+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:37.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:37.181+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:37.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:37.094+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:37.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:37.094+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:37.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:37.094+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:37.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:37.094+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:37.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:37.546+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:37.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:37.546+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:37.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:37.546+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:37.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:37.546+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:38.435 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:18:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:18:38.208+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 26027 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:18:38.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:38.208+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:38.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:38.208+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:38.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:38.208+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:38.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:38.209+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:38.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:38.088+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:38.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:38.088+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:38.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:38.088+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:38.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:38.088+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:38.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:38.519+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:38.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:38.519+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:38.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:38.519+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:38.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:38.520+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:39.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:39.121+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:39.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:39.121+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:39.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:39.121+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:39.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:39.121+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:39.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:39.172+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:39.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:39.172+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:39.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:39.172+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:39.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:39.172+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:39.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:39.519+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:39.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:39.519+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:39.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:39.519+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:39.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:39.519+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:40.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:40.082+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:40.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:40.082+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:40.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:40.082+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:40.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:40.082+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:40.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:40.221+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:40.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:40.221+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:40.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:40.221+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:40.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:40.221+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:40.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:40.533+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:40.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:40.533+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:40.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:40.533+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:40.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:40.534+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:41.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:41.036+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:41.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:41.036+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:41.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:41.036+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:41.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:41.036+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:41.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:41.229+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:41.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:41.229+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:41.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:41.229+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:41.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:41.229+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:41.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:41.542+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:41.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:41.542+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:41.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:41.542+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:41.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:41.543+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:42.271 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:42.084+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:42.271 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:42.084+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:42.271 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:42.084+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:42.272 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:42.084+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:42.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:42.501+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:42.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:42.501+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:42.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:42.501+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:42.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:42.501+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:42.687 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:42.276+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:42.687 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:42.276+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:42.687 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:42.276+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:42.687 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:42.276+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:43.435 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:18:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:18:43.212+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 26117 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:18:43.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:43.254+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:43.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:43.254+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:43.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:43.254+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:43.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:43.255+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:43.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:43.125+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:43.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:43.125+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:43.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:43.125+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:43.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:43.125+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:43.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:43.510+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:43.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:43.510+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:43.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:43.510+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:43.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:43.511+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:44.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:44.115+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:44.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:44.115+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:44.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:44.115+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:44.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:44.115+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:44.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:44.245+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:44.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:44.245+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:44.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:44.245+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:44.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:44.245+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:44.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:44.462+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:44.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:44.462+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:44.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:44.462+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:44.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:44.462+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:45.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:45.259+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:45.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:45.259+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:45.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:45.259+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:45.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:45.259+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:45.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:45.133+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:45.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:45.133+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:45.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:45.133+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:45.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:45.134+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:45.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:45.484+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:45.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:45.484+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:45.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:45.484+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:45.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:45.484+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:46.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:46.145+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:46.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:46.145+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:46.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:46.145+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:46.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:46.145+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:46.432 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:46.247+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:46.432 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:46.247+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:46.433 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:46.247+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:46.433 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:46.247+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:46.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:46.437+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:46.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:46.437+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:46.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:46.437+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:46.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:46.437+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:47.469 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:47.193+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:47.469 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:47.193+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:47.469 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:47.193+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:47.469 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:47.193+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:47.470 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:47.199+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:47.470 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:47.199+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:47.470 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:47.199+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:47.470 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:47.199+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:47.736 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:47.474+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:47.736 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:47.474+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:47.736 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:47.474+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:47.736 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:47.474+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:48.508 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:18:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:18:48.217+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 26208 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:18:48.508 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:48.206+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:48.509 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:48.206+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:48.509 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:48.206+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:48.509 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:48.206+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:48.509 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:48.215+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:48.509 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:48.215+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:48.509 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:48.215+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:48.510 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:48.215+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:48.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:48.512+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:48.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:48.512+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:48.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:48.512+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:48.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:48.513+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:49.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:49.189+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:49.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:49.189+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:49.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:49.189+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:49.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:49.190+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:49.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:49.178+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:49.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:49.178+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:49.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:49.178+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:49.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:49.179+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:49.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:49.530+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:49.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:49.530+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:49.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:49.530+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:49.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:49.530+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:50.480 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:50.199+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:50.480 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:50.199+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:50.480 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:50.199+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:50.481 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:50.199+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:50.481 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:50.197+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:50.481 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:50.197+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:50.481 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:50.197+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:50.482 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:50.197+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:50.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:50.502+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:50.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:50.502+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:50.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:50.502+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:50.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:50.502+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:51.495 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:51.198+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:51.495 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:51.198+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:51.496 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:51.198+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:51.496 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:51.198+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:51.496 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:51.218+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:51.496 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:51.218+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:51.496 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:51.218+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:51.496 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:51.218+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:51.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:51.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:51.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:51.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:51.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:51.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:51.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:51.500+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:52.261 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:52.183+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:52.261 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:52.183+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:52.261 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:52.183+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:52.261 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:52.183+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:52.514 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:52.265+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:52.514 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:52.265+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:52.514 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:52.265+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:52.514 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:52.265+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:52.778 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:52.519+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:52.779 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:52.519+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:52.779 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:52.519+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:52.779 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:52.519+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:53.435 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:18:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:18:53.220+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 26297 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:18:53.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:53.265+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:53.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:53.265+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:53.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:53.265+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:53.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:53.265+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:53.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:53.151+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:53.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:53.151+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:53.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:53.151+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:53.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:53.151+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:53.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:53.480+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:53.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:53.480+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:53.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:53.480+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:53.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:53.480+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:54.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:54.260+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:54.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:54.260+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:54.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:54.260+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:54.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:54.260+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:54.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:54.180+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:54.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:54.180+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:54.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:54.180+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:54.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:54.180+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:54.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:54.506+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:54.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:54.506+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:54.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:54.506+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:54.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:54.506+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:55.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:55.221+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:55.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:55.221+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:55.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:55.221+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:55.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:55.221+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:55.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:55.161+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:55.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:55.161+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:55.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:55.161+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:55.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:55.161+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:55.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:55.494+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:55.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:55.494+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:55.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:55.494+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:55.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:55.494+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:56.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:56.222+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:56.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:56.222+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:56.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:56.222+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:56.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:56.222+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:56.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:56.149+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:56.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:56.149+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:56.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:56.149+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:56.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:56.149+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:56.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:56.529+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:56.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:56.529+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:56.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:56.529+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:56.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:56.529+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:57.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:57.205+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:57.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:57.205+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:57.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:57.205+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:57.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:57.205+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:57.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:57.137+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:57.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:57.137+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:57.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:57.137+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:57.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:57.137+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:57.820 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:57.523+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:57.821 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:57.523+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:57.821 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:57.523+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:57.821 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:57.523+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:58.435 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:18:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:18:58.225+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 26387 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:18:58.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:58.183+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:58.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:58.183+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:58.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:58.183+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:58.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:58.184+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:58.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:58.254+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:58.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:58.254+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:58.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:58.254+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:58.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:58.254+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:58.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:58.556+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:58.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:58.556+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:58.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:58.556+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:58.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:58.556+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:59.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:59.175+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:59.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:59.175+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:59.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:59.175+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:18:59.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:18:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:18:59.175+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:18:59.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:59.229+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:59.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:59.229+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:59.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:59.229+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:18:59.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:18:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:18:59.229+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:18:59.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:59.562+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:59.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:59.562+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:59.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:59.562+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:18:59.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:18:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:18:59.562+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:00.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:00.140+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:00.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:00.140+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:00.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:00.140+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:00.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:00.141+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:00.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:00.261+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:00.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:00.261+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:00.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:00.261+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:00.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:00.261+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:00.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:00.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:00.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:00.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:00.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:00.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:00.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:00.568+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:01.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:01.094+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:01.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:01.094+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:01.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:01.094+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:01.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:01.094+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:01.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:01.231+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:01.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:01.231+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:01.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:01.231+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:01.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:01.232+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:01.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:01.574+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:01.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:01.574+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:01.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:01.574+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:01.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:01.574+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:02.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:02.090+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:02.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:02.090+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:02.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:02.090+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:02.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:02.090+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:02.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:02.209+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:02.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:02.209+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:02.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:02.209+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:02.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:02.209+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:02.862 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:02.558+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:02.863 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:02.558+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:02.863 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:02.558+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:02.863 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:02.559+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:03.436 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:19:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:19:03.229+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 26477 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:19:03.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:03.217+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:03.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:03.217+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:03.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:03.217+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:03.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:03.217+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:03.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:03.128+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:03.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:03.128+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:03.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:03.128+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:03.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:03.128+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:03.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:03.564+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:03.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:03.564+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:03.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:03.564+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:03.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:03.564+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:04.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:04.232+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:04.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:04.232+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:04.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:04.232+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:04.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:04.232+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:04.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:04.164+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:04.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:04.164+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:04.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:04.164+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:04.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:04.164+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:04.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:04.551+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:04.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:04.551+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:04.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:04.551+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:04.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:04.551+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:05.585 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:05.211+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:05.585 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:05.211+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:05.586 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:05.211+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:05.586 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:05.211+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:05.586 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:05.197+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:05.586 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:05.197+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:05.586 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:05.197+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:05.586 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:05.197+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:05.840 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:05.589+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:05.840 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:05.589+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:05.840 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:05.589+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:05.840 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:05.590+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:06.567 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:06.214+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:06.567 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:06.214+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:06.567 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:06.214+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:06.567 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:06.214+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:06.567 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:06.228+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:06.567 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:06.228+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:06.568 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:06.228+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:06.568 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:06.228+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:06.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:06.570+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:06.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:06.571+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:06.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:06.571+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:06.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:06.571+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:07.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:07.181+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:07.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:07.181+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:07.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:07.181+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:07.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:07.181+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:07.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:07.216+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:07.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:07.216+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:07.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:07.216+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:07.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:07.216+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:07.905 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:07.596+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:07.905 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:07.596+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:07.906 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:07.596+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:07.906 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:07.596+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:08.612 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:19:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:19:08.234+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 26568 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:19:08.613 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:08.211+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:08.613 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:08.211+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:08.613 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:08.211+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:08.613 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:08.211+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:08.613 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:08.213+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:08.613 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:08.213+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:08.614 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:08.213+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:08.614 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:08.213+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:08.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:08.616+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:08.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:08.616+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:08.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:08.616+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:08.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:08.617+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:09.584 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:09.249+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:09.585 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:09.249+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:09.585 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:09.249+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:09.585 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:09.250+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:09.585 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:09.241+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:09.586 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:09.241+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:09.586 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:09.241+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:09.586 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:09.241+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:09.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:09.589+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:09.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:09.589+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:09.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:09.589+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:09.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:09.589+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:10.595 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:10.235+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:10.595 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:10.235+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:10.595 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:10.235+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:10.595 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:10.236+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:10.596 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:10.202+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:10.596 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:10.202+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:10.596 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:10.202+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:10.596 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:10.202+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:10.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:10.599+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:10.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:10.599+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:10.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:10.599+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:10.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:10.599+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:11.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:11.231+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:11.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:11.231+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:11.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:11.231+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:11.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:11.231+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:11.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:11.167+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:11.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:11.167+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:11.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:11.167+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:11.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:11.167+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:11.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:11.628+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:11.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:11.628+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:11.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:11.628+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:11.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:11.628+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:12.237 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:12.186+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:12.238 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:12.186+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:12.238 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:12.186+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:12.238 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:12.187+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:12.619 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:12.241+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:12.620 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:12.241+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:12.620 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:12.241+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:12.620 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:12.242+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:12.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:12.623+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:12.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:12.623+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:12.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:12.623+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:12.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:12.624+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:13.234 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:13.155+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:13.235 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:13.155+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:13.235 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:13.155+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:13.235 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:13.155+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:13.628 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:19:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:19:13.238+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 26658 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:19:13.629 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:13.280+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:13.629 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:13.280+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:13.629 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:13.280+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:13.629 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:13.281+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:13.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:13.654+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:13.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:13.654+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:13.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:13.654+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:13.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:13.654+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:14.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:14.241+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:14.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:14.241+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:14.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:14.241+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:14.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:14.241+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:14.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:14.177+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:14.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:14.177+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:14.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:14.177+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:14.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:14.178+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:14.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:14.683+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:14.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:14.683+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:14.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:14.683+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:14.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:14.684+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:15.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:15.258+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:15.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:15.258+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:15.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:15.258+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:15.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:15.259+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:15.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:15.198+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:15.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:15.198+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:15.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:15.198+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:15.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:15.198+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:15.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:15.652+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:15.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:15.652+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:15.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:15.652+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:15.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:15.652+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:16.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:16.227+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:16.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:16.227+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:16.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:16.227+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:16.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:16.227+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:16.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:16.156+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:16.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:16.156+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:16.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:16.156+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:16.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:16.156+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:16.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:16.611+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:16.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:16.611+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:16.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:16.611+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:16.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:16.612+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:17.561 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:17.237+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:17.561 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:17.237+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:17.561 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:17.237+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:17.561 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:17.237+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:17.562 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:17.198+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:17.562 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:17.198+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:17.562 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:17.198+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:17.562 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:17.198+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:17.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:17.565+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:17.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:17.565+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:17.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:17.565+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:17.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:17.565+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:18.250 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:19:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:19:18.242+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 26749 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:19:18.251 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:18.190+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:18.251 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:18.190+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:18.251 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:18.190+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:18.251 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:18.190+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:18.251 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:18.192+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:18.251 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:18.192+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:18.252 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:18.192+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:18.252 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:18.193+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:18.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:18.614+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:18.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:18.614+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:18.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:18.614+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:18.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:18.614+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:19.566 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:19.207+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:19.566 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:19.207+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:19.566 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:19.207+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:19.566 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:19.207+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:19.567 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:19.236+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:19.567 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:19.236+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:19.567 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:19.236+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:19.567 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:19.237+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:19.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:19.570+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:19.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:19.570+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:19.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:19.570+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:19.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:19.570+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:20.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:20.164+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:20.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:20.164+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:20.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:20.164+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:20.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:20.164+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:20.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:20.275+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:20.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:20.275+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:20.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:20.275+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:20.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:20.275+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:20.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:20.572+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:20.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:20.572+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:20.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:20.572+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:20.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:20.572+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:21.528 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:21.275+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:21.528 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:21.275+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:21.528 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:21.275+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:21.528 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:21.275+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:21.529 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:21.211+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:21.529 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:21.211+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:21.529 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:21.211+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:21.529 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:21.212+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:21.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:21.532+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:21.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:21.532+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:21.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:21.532+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:21.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:21.532+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:22.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:22.204+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:22.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:22.204+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:22.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:22.204+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:22.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:22.204+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:22.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:22.253+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:22.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:22.253+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:22.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:22.253+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:22.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:22.253+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:22.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:22.567+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:22.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:22.567+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:22.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:22.567+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:22.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:22.567+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:23.299 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:19:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:19:23.247+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 26840 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:19:23.300 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:23.164+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:23.300 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:23.164+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:23.300 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:23.164+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:23.300 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:23.164+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:23.598 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:23.303+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:23.599 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:23.303+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:23.599 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:23.303+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:23.599 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:23.304+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:23.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:23.603+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:23.981 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:23.603+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:23.981 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:23.603+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:23.981 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:23.603+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:24.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:24.307+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:24.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:24.307+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:24.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:24.307+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:24.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:24.307+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:24.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:24.115+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:24.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:24.115+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:24.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:24.115+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:24.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:24.115+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:24.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:24.572+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:24.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:24.572+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:24.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:24.572+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:24.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:24.572+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:25.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:25.271+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:25.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:25.271+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:25.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:25.271+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:25.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:25.271+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:25.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:25.111+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:25.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:25.111+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:25.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:25.111+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:25.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:25.112+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:25.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:25.616+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:25.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:25.616+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:25.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:25.616+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:25.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:25.616+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:26.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:26.312+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:26.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:26.312+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:26.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:26.312+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:26.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:26.312+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:26.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:26.080+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:26.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:26.080+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:26.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:26.080+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:26.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:26.080+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:26.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:26.629+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:26.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:26.629+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:26.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:26.629+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:26.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:26.629+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:27.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:27.278+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:27.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:27.278+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:27.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:27.278+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:27.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:27.278+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:27.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:27.060+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:27.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:27.060+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:27.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:27.060+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:27.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:27.061+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:27.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:27.595+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:27.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:27.595+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:27.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:27.595+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:27.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:27.596+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:28.371 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:19:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:19:28.251+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 26931 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:19:28.372 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:28.319+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:28.372 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:28.319+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:28.372 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:28.319+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:28.372 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:28.319+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:28.372 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:28.101+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:28.373 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:28.101+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:28.373 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:28.101+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:28.373 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:28.101+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:28.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:28.607+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:28.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:28.607+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:28.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:28.607+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:28.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:28.607+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:29.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:29.082+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:29.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:29.082+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:29.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:29.082+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:29.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:29.083+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:29.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:29.284+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:29.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:29.284+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:29.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:29.284+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:29.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:29.284+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:29.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:29.628+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:29.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:29.628+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:29.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:29.628+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:29.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:29.628+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:30.416 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:30.241+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:30.416 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:30.241+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:30.416 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:30.241+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:30.416 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:30.242+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:30.417 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:30.065+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:30.417 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:30.065+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:30.417 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:30.065+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:30.417 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:30.065+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:30.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:30.661+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:30.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:30.661+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:30.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:30.661+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:30.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:30.661+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:31.435 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:31.070+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:31.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:31.070+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:31.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:31.070+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:31.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:31.070+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:31.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:31.287+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:31.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:31.287+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:31.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:31.287+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:31.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:31.287+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:31.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:31.643+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:31.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:31.643+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:31.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:31.643+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:31.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:31.643+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:32.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:32.028+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:32.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:32.028+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:32.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:32.028+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:32.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:32.028+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:32.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:32.252+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:32.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:32.252+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:32.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:32.252+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:32.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:32.252+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:32.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:32.615+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:32.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:32.615+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:32.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:32.615+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:32.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:32.615+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:33.250 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:32.984+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:33.250 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:32.984+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:33.250 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:32.984+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:33.250 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:32.984+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:33.585 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:19:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:19:33.255+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 27021 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:19:33.585 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:33.253+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:33.586 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:33.253+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:33.586 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:33.253+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:33.586 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:33.254+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:33.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:33.589+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:33.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:33.589+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:33.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:33.589+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:33.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:33.589+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:34.301 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:33.978+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:34.302 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:33.978+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:34.302 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:33.978+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:34.302 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:33.978+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:34.564 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:34.306+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:34.564 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:34.306+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:34.564 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:34.306+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:34.565 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:34.306+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:34.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:34.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:34.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:34.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:34.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:34.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:34.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:34.568+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:35.336 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:34.981+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:35.336 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:34.981+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:35.336 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:34.981+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:35.336 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:34.981+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:35.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:35.531+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:35.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:35.531+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:35.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:35.531+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:35.687 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:35.532+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:35.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:35.340+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:35.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:35.340+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:35.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:35.340+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:35.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:35.340+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:36.314 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:35.958+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:36.314 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:35.958+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:36.314 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:35.958+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:36.315 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:35.958+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:36.686 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:36.317+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:36.686 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:36.317+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:36.686 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:36.318+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:36.686 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:36.318+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:36.687 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:36.492+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:36.687 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:36.492+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:36.687 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:36.492+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:36.687 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:36.492+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:37.356 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:36.996+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:37.356 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:36.996+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:37.356 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:36.996+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:37.357 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:36.996+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:37.685 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:37.529+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:37.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:37.529+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:37.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:37.529+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:37.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:37.529+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:37.686 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:37.360+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:37.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:37.360+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:37.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:37.360+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:37.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:37.360+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:38.256 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:38.009+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:38.256 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:38.009+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:38.256 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:38.009+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:38.257 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:38.010+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:38.522 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:19:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:19:38.260+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 27112 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:19:38.522 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:38.349+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:38.522 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:38.349+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:38.523 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:38.349+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:38.523 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:38.349+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:38.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:38.526+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:38.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:38.526+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:38.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:38.526+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:38.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:38.526+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:39.340 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:38.968+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:39.340 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:38.968+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:39.341 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:38.968+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:39.341 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:38.969+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:39.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:39.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:39.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:39.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:39.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:39.568+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:39.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:39.568+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:39.686 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:39.344+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:39.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:39.344+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:39.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:39.344+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:39.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:39.344+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:40.342 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:39.973+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:40.342 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:39.973+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:40.343 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:39.973+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:40.343 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:39.974+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:40.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:40.565+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:40.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:40.565+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:40.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:40.565+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:40.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:40.565+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:40.686 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:40.347+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:40.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:40.347+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:40.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:40.347+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:40.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:40.347+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:41.322 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:41.009+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:41.322 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:41.009+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:41.322 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:41.009+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:41.322 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:41.009+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:41.572 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:41.325+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:41.573 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:41.325+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:41.573 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:41.325+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:41.573 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:41.326+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:41.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:41.576+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:41.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:41.576+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:41.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:41.576+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:41.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:41.577+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:42.016 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:42.018+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:42.016 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:42.018+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:42.017 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:42.018+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:42.017 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:42.019+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:42.571 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:42.312+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:42.571 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:42.312+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:42.571 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:42.312+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:42.571 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:42.312+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:42.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:42.574+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:42.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:42.574+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:42.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:42.574+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:42.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:42.574+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:43.260 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:42.988+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:43.261 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:42.988+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:43.261 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:42.988+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:43.261 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:42.988+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:43.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:19:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:19:43.263+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 27204 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:19:43.553 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:43.315+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:43.554 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:43.315+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:43.554 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:43.315+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:43.554 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:43.315+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:43.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:43.557+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:43.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:43.557+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:43.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:43.557+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:43.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:43.557+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:44.312 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:43.952+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:44.312 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:43.952+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:44.312 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:43.952+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:44.312 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:43.952+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:44.597 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:44.315+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:44.597 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:44.315+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:44.597 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:44.315+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:44.597 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:44.315+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:44.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:44.600+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:44.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:44.600+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:44.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:44.600+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:44.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:44.600+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:45.326 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:45.000+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:45.326 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:45.000+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:45.326 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:45.000+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:45.327 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:45.000+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:45.597 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:45.330+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:45.598 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:45.330+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:45.598 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:45.330+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:45.598 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:45.330+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:45.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:45.601+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:45.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:45.601+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:45.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:45.601+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:45.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:45.602+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:46.310 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:45.975+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:46.311 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:45.975+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:46.311 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:45.975+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:46.311 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:45.975+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:46.566 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:46.314+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:46.566 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:46.314+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:46.566 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:46.314+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:46.566 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:46.314+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:46.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:46.569+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:46.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:46.569+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:46.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:46.569+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:46.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:46.570+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:47.327 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:46.981+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:47.327 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:46.981+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:47.327 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:46.981+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:47.327 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:46.981+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:47.579 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:47.330+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:47.579 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:47.330+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:47.579 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:47.330+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:47.580 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:47.331+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:47.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:47.583+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:47.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:47.583+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:47.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:47.583+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:47.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:47.583+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:48.265 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:48.009+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:48.265 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:48.009+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:48.265 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:48.009+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:48.265 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:48.010+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:48.615 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:19:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:19:48.268+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 27294 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:19:48.616 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:48.349+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:48.616 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:48.349+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:48.616 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:48.349+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:48.616 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:48.349+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:48.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:48.619+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:48.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:48.619+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:48.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:48.619+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:48.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:48.620+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:49.334 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:48.961+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:49.334 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:48.961+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:49.334 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:48.961+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:49.334 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:48.962+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:49.623 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:49.337+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:49.624 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:49.337+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:49.624 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:49.337+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:49.624 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:49.338+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:49.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:49.627+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:49.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:49.627+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:49.931 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:49.627+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:49.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:49.628+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:50.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:49.934+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:50.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:49.934+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:50.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:49.934+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:50.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:49.935+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:50.588 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:50.352+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:50.588 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:50.352+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:50.589 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:50.352+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:50.589 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:50.352+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:50.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:50.592+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:50.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:50.592+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:50.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:50.592+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:50.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:50.592+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:51.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:50.938+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:51.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:50.938+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:51.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:50.938+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:51.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:50.938+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:51.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:51.579+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:51.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:51.579+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:51.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:51.579+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:51.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:51.579+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:51.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:51.390+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:51.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:51.390+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:51.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:51.390+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:51.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:51.390+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:52.185 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:51.976+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:52.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:51.976+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:52.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:51.976+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:52.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:51.976+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:52.595 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:52.398+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:52.595 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:52.398+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:52.595 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:52.398+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:52.595 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:52.398+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:52.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:52.598+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:52.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:52.598+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:52.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:52.598+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:52.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:52.599+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:53.270 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:53.002+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:53.270 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:53.002+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:53.270 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:53.002+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:53.270 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:53.002+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:53.624 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:19:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:19:53.273+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 27385 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:19:53.624 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:53.393+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:53.625 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:53.393+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:53.625 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:53.393+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:53.625 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:53.393+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:53.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:53.628+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:53.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:53.628+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:53.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:53.628+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:53.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:53.628+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:54.348 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:54.028+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:54.348 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:54.028+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:54.348 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:54.028+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:54.348 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:54.029+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:54.638 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:54.351+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:54.638 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:54.351+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:54.638 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:54.351+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:54.639 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:54.351+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:54.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:54.642+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:55.069 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:54.642+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:55.070 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:54.642+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:55.070 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:54.642+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:55.395 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:54.983+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:55.395 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:54.983+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:55.395 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:54.983+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:55.395 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:54.984+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:55.663 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:55.398+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:55.664 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:55.398+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:55.664 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:55.398+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:55.664 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:55.398+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:55.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:55.667+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:55.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:55.667+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:55.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:55.667+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:55.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:55.667+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:56.400 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:55.954+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:56.400 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:55.954+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:56.400 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:55.954+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:56.400 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:55.954+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:56.651 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:56.403+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:56.651 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:56.403+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:56.652 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:56.403+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:56.652 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:56.404+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:56.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:56.655+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:56.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:56.655+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:56.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:56.655+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:56.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:56.655+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:57.364 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:56.954+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:57.364 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:56.954+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:57.365 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:56.954+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:57.365 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:56.954+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:57.621 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:57.368+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:57.621 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:57.368+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:57.621 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:57.368+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:57.621 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:57.368+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:57.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:57.624+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:57.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:57.624+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:57.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:57.624+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:57.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:57.624+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:58.274 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:57.994+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:58.274 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:57.994+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:58.275 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:57.994+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:58.275 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:57.994+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:58.582 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:19:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:19:58.277+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 27475 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:19:58.582 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:58.326+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:58.583 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:58.326+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:58.583 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:58.326+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:58.583 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:58.326+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:58.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:58.585+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:58.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:58.585+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:58.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:58.585+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:58.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:58.586+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:59.344 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:59.005+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:59.344 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:59.005+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:59.344 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:59.005+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:19:59.344 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:59.006+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:19:59.686 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:59.347+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:59.686 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:59.347+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:59.686 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:59.347+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:19:59.686 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:19:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:19:59.348+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:19:59.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:59.574+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:59.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:59.574+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:59.687 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:59.574+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:19:59.687 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:19:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:19:59.575+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:00.323 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:59.986+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:00.323 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:59.986+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:00.323 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:59.986+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:00.323 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:19:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:19:59.987+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:00.611 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:00.327+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:00.611 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:00.327+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:00.612 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:00.327+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:00.612 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:00.327+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:00.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:00.615+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:00.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:00.615+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:00.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:00.615+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:00.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:00.615+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:01.286 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:00.962+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:01.287 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:00.962+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:01.287 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:00.962+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:01.287 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:00.962+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:01.589 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:01.290+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:01.590 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:01.290+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:01.590 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:01.290+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:01.590 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:01.290+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:01.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:01.593+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:01.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:01.593+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:01.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:01.593+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:01.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:01.593+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:02.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:01.935+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:02.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:01.935+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:02.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:01.935+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:02.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:01.936+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:02.571 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:02.293+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:02.572 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:02.293+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:02.572 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:02.293+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:02.572 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:02.293+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:02.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:02.575+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:02.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:02.575+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:02.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:02.575+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:02.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:02.575+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:03.251 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:02.956+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:03.251 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:02.956+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:03.252 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:02.956+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:03.252 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:02.956+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:03.528 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:20:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:20:03.282+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 27567 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:20:03.529 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:03.255+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:03.529 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:03.255+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:03.529 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:03.255+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:03.529 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:03.255+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:03.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:03.532+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:03.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:03.532+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:03.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:03.532+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:03.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:03.532+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:04.209 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:03.947+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:04.209 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:03.947+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:04.209 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:03.947+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:04.209 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:03.948+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:04.515 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:04.294+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:04.515 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:04.294+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:04.516 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:04.294+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:04.516 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:04.294+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:04.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:04.519+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:04.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:04.519+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:04.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:04.519+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:04.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:04.519+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:05.258 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:04.971+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:05.258 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:04.971+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:05.259 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:04.971+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:05.259 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:04.972+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:05.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:05.486+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:05.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:05.486+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:05.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:05.486+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:05.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:05.486+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:05.686 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:05.262+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:05.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:05.262+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:05.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:05.262+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:05.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:05.262+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:06.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:06.010+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:06.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:06.010+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:06.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:06.010+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:06.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:06.010+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:06.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:06.250+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:06.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:06.250+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:06.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:06.250+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:06.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:06.250+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:06.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:06.490+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:06.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:06.490+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:06.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:06.490+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:06.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:06.491+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:07.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:07.037+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:07.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:07.037+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:07.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:07.037+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:07.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:07.037+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:07.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:07.207+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:07.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:07.207+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:07.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:07.207+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:07.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:07.207+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:07.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:07.453+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:07.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:07.453+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:07.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:07.453+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:07.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:07.454+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:08.407 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:20:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:20:08.287+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 27657 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:20:08.408 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:08.063+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:08.408 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:08.063+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:08.408 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:08.063+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:08.408 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:08.063+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:08.409 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:08.254+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:08.409 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:08.254+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:08.409 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:08.254+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:08.409 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:08.254+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:08.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:08.471+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:08.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:08.471+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:08.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:08.471+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:08.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:08.471+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:09.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:09.037+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:09.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:09.037+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:09.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:09.037+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:09.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:09.037+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:09.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:09.269+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:09.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:09.269+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:09.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:09.269+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:09.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:09.270+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:09.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:09.494+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:09.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:09.494+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:09.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:09.494+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:09.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:09.494+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:10.353 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:10.055+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:10.353 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:10.055+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:10.353 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:10.055+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:10.353 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:10.056+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:10.353 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:10.223+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:10.354 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:10.223+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:10.354 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:10.223+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:10.354 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:10.223+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:10.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:10.532+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:10.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:10.532+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:10.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:10.532+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:10.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:10.532+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:11.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:11.020+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:11.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:11.020+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:11.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:11.020+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:11.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:11.020+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:11.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:11.225+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:11.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:11.225+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:11.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:11.225+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:11.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:11.225+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:11.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:11.567+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:11.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:11.567+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:11.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:11.567+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:11.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:11.567+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:12.268 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:12.027+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:12.268 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:12.027+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:12.269 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:12.027+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:12.269 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:12.027+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:12.523 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:12.272+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:12.523 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:12.272+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:12.523 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:12.272+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:12.524 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:12.272+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:12.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:12.527+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:12.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:12.527+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:12.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:12.527+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:12.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:12.527+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:13.414 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:20:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:20:13.292+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 27749 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:20:13.414 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:13.072+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:13.414 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:13.072+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:13.414 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:13.072+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:13.415 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:13.072+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:13.415 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:13.237+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:13.415 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:13.237+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:13.415 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:13.237+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:13.415 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:13.237+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:13.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:13.550+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:13.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:13.550+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:13.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:13.550+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:13.686 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:13.550+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:14.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:14.024+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:14.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:14.024+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:14.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:14.024+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:14.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:14.024+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:14.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:14.253+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:14.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:14.253+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:14.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:14.253+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:14.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:14.253+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:14.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:14.510+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:14.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:14.510+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:14.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:14.510+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:14.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:14.510+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:15.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:15.069+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:15.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:15.069+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:15.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:15.069+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:15.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:15.069+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:15.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:15.249+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:15.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:15.249+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:15.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:15.249+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:15.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:15.249+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:15.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:15.546+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:15.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:15.546+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:15.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:15.546+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:15.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:15.546+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:16.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:16.070+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:16.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:16.070+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:16.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:16.070+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:16.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:16.070+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:16.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:16.290+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:16.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:16.290+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:16.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:16.290+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:16.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:16.290+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:16.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:16.597+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:16.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:16.597+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:16.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:16.597+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:16.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:16.597+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:17.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:17.048+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:17.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:17.048+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:17.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:17.048+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:17.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:17.048+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:17.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:17.244+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:17.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:17.244+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:17.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:17.244+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:17.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:17.244+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:17.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:17.602+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:17.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:17.602+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:17.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:17.602+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:17.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:17.602+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:18.288 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:18.040+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:18.288 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:18.040+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:18.289 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:18.040+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:18.289 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:18.040+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:18.576 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:20:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:20:18.296+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 27838 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:20:18.577 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:18.291+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:18.577 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:18.291+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:18.577 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:18.291+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:18.577 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:18.292+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:18.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:18.580+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:18.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:18.580+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:18.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:18.580+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:18.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:18.580+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:19.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:19.243+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:19.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:19.243+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:19.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:19.243+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:19.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:19.243+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:19.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:19.041+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:19.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:19.041+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:19.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:19.041+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:19.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:19.041+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:19.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:19.610+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:19.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:19.610+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:19.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:19.610+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:19.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:19.611+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:20.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:20.288+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:20.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:20.288+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:20.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:20.288+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:20.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:20.288+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:20.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:20.061+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:20.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:20.061+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:20.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:20.061+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:20.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:20.061+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:20.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:20.641+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:20.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:20.641+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:20.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:20.641+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:20.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:20.642+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:21.310 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:21.014+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:21.311 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:21.014+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:21.311 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:21.014+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:21.311 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:21.014+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:21.663 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:21.314+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:21.663 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:21.314+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:21.664 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:21.314+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:21.664 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:21.314+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:21.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:21.667+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:21.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:21.667+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:21.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:21.667+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:21.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:21.667+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:22.353 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:21.982+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:22.353 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:21.982+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:22.353 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:21.982+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:22.353 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:21.982+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:22.640 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:22.356+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:22.641 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:22.356+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:22.641 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:22.356+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:22.641 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:22.356+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:22.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:22.690+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:22.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:22.690+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:22.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:22.690+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:22.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:22.691+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:23.297 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:22.995+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:23.298 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:22.995+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:23.298 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:22.995+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:23.298 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:22.995+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:23.633 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:20:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:20:23.300+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 27931 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:20:23.634 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:23.398+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:23.634 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:23.398+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:23.634 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:23.398+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:23.634 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:23.398+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:23.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:23.683+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:23.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:23.683+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:23.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:23.683+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:23.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:23.683+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:24.366 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:24.023+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:24.366 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:24.023+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:24.366 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:24.023+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:24.367 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:24.023+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:24.686 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:24.370+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:24.686 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:24.370+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:24.686 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:24.370+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:24.686 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:24.370+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:25.047 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:24.699+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:25.047 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:24.699+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:25.047 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:24.699+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:25.047 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:24.700+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:25.351 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:25.050+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:25.352 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:25.050+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:25.352 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:25.050+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:25.352 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:25.050+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:25.686 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:25.355+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:25.686 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:25.355+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:25.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:25.355+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:25.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:25.355+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:26.078 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:25.738+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:26.078 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:25.738+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:26.079 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:25.738+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:26.079 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:25.738+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:26.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:26.082+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:26.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:26.082+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:26.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:26.082+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:26.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:26.082+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:26.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:26.330+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:26.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:26.330+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:26.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:26.330+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:26.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:26.330+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:27.075 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:26.737+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:27.075 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:26.737+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:27.075 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:26.737+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:27.075 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:26.738+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:27.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:27.078+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:27.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:27.078+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:27.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:27.078+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:27.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:27.078+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:27.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:27.321+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:27.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:27.321+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:27.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:27.321+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:27.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:27.321+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:28.029 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:27.714+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:28.029 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:27.714+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:28.029 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:27.714+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:28.029 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:27.714+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:28.302 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:28.032+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:28.302 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:28.032+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:28.302 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:28.032+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:28.303 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:28.032+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:28.303 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:28.275+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:28.303 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:28.275+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:28.303 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:28.275+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:28.303 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:28.275+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:28.583 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:20:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:20:28.305+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 28020 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:20:28.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:28.680+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:28.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:28.680+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:28.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:28.680+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:28.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:28.680+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:29.306 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:29.057+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:29.307 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:29.057+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:29.307 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:29.057+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:29.307 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:29.058+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:29.659 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:29.310+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:29.661 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:29.310+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:29.661 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:29.310+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:29.662 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:29.310+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:29.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:29.663+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:29.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:29.663+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:29.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:29.663+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:29.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:29.663+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:30.325 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:30.009+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:30.326 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:30.009+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:30.326 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:30.009+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:30.326 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:30.009+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:30.667 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:30.329+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:30.667 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:30.329+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:30.667 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:30.329+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:30.668 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:30.329+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:30.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:30.670+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:30.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:30.670+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:30.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:30.670+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:30.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:30.671+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:31.313 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:31.041+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:31.313 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:31.041+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:31.314 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:31.041+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:31.314 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:31.041+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:31.669 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:31.317+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:31.669 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:31.317+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:31.669 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:31.317+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:31.670 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:31.317+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:31.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:31.672+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:31.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:31.672+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:31.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:31.672+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:31.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:31.672+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:32.314 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:32.087+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:32.314 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:32.087+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:32.314 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:32.087+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:32.314 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:32.087+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:32.686 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:32.317+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:32.686 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:32.317+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:32.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:32.317+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:32.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:32.317+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:33.131 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:32.709+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:33.131 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:32.709+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:33.131 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:32.709+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:33.132 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:32.709+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:33.433 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:20:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:20:33.309+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 28112 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:20:33.434 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:33.132+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:33.434 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:33.132+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:33.434 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:33.132+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:33.434 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:33.133+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:33.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:33.325+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:33.434 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:33.325+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:33.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:33.325+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:33.435 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:33.326+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:34.101 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:33.731+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:34.101 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:33.731+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:34.101 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:33.731+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:34.101 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:33.731+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:34.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:34.104+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:34.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:34.104+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:34.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:34.104+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:34.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:34.104+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:34.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:34.295+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:34.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:34.295+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:34.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:34.295+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:34.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:34.295+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:35.065 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:34.718+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:35.065 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:34.718+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:35.065 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:34.718+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:35.065 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:34.719+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:35.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:35.068+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:35.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:35.068+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:35.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:35.068+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:35.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:35.068+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:35.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:35.253+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:35.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:35.253+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:35.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:35.253+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:35.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:35.253+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:36.068 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:35.757+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:36.069 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:35.757+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:36.069 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:35.757+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:36.069 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:35.757+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:36.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:36.071+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:36.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:36.071+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:36.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:36.071+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:36.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:36.072+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:36.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:36.294+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:36.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:36.294+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:36.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:36.294+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:36.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:36.294+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:37.082 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:36.749+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:37.082 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:36.749+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:37.082 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:36.749+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:37.082 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:36.749+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:37.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:37.085+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:37.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:37.085+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:37.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:37.085+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:37.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:37.085+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:37.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:37.254+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:37.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:37.254+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:37.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:37.254+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:37.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:37.254+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:38.090 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:37.741+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:38.091 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:37.741+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:38.091 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:37.741+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:38.091 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:37.742+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:38.431 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:20:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:20:38.314+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 28201 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:20:38.432 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:38.093+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:38.432 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:38.093+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:38.432 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:38.093+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:38.432 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:38.094+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:38.432 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:38.282+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:38.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:38.282+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:38.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:38.282+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:38.433 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:38.282+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:39.131 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:38.743+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:39.131 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:38.743+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:39.131 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:38.743+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:39.132 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:38.744+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:39.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:39.134+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:39.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:39.134+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:39.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:39.134+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:39.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:39.135+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:39.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:39.319+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:39.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:39.319+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:39.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:39.319+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:39.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:39.320+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:40.166 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:39.731+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:40.166 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:39.731+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:40.166 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:39.731+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:40.166 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:39.732+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:40.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:40.169+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:40.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:40.169+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:40.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:40.169+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:40.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:40.169+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:40.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:40.310+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:40.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:40.310+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:40.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:40.310+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:40.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:40.310+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:41.073 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:40.757+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:41.073 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:40.757+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:41.073 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:40.757+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:41.073 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:40.758+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:41.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:41.187+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:41.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:41.187+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:41.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:41.187+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:41.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:41.188+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:41.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:41.273+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:41.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:41.273+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:41.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:41.273+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:41.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:41.273+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:42.177 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:41.772+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:42.177 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:41.772+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:42.177 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:41.772+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:42.177 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:41.772+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:42.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:42.180+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:42.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:42.180+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:42.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:42.180+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:42.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:42.180+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:42.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:42.246+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:42.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:42.246+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:42.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:42.246+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:42.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:42.246+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:43.186 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:42.778+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:43.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:42.778+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:43.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:42.778+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:43.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:42.779+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:43.636 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:20:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:20:43.318+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 28294 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:20:43.636 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:43.275+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:43.637 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:43.275+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:43.637 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:43.275+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:43.637 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:43.275+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:43.637 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:43.193+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:43.637 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:43.193+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:43.637 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:43.193+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:43.638 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:43.194+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:43.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:43.805+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:43.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:43.805+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:43.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:43.805+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:43.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:43.805+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:44.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:44.299+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:44.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:44.299+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:44.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:44.299+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:44.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:44.299+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:44.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:44.171+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:44.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:44.171+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:44.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:44.171+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:44.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:44.172+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:45.120 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:44.785+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:45.120 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:44.785+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:45.121 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:44.785+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:45.121 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:44.786+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:45.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:45.307+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:45.439 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:45.307+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:45.439 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:45.308+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:45.439 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:45.308+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:45.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:45.123+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:45.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:45.123+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:45.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:45.123+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:45.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:45.124+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:46.082 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:45.761+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:46.082 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:45.761+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:46.082 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:45.761+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:46.082 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:45.761+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:46.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:46.085+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:46.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:46.085+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:46.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:46.085+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:46.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:46.085+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:46.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:46.324+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:46.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:46.324+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:46.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:46.324+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:46.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:46.324+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:47.079 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:46.805+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:47.079 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:46.805+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:47.079 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:46.805+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:47.080 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:46.806+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:47.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:47.082+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:47.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:47.082+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:47.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:47.082+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:47.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:47.082+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:47.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:47.290+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:47.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:47.290+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:47.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:47.290+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:47.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:47.291+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:48.101 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:47.834+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:48.102 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:47.834+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:48.102 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:47.834+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:48.102 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:47.835+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:48.436 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:20:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:20:48.323+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 28383 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:20:48.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:48.253+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:48.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:48.253+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:48.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:48.253+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:48.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:48.253+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:48.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:48.102+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:48.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:48.102+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:48.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:48.102+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:48.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:48.102+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:49.051 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:48.786+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:49.051 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:48.786+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:49.051 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:48.786+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:49.052 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:48.787+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:49.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:49.054+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:49.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:49.054+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:49.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:49.054+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:49.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:49.055+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:49.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:49.204+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:49.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:49.204+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:49.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:49.204+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:49.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:49.204+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:50.068 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:49.820+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:50.068 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:49.820+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:50.068 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:49.820+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:50.068 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:49.820+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:50.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:50.071+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:50.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:50.071+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:50.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:50.071+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:50.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:50.071+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:50.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:50.167+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:50.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:50.167+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:50.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:50.167+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:50.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:50.167+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:51.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:51.029+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:51.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:51.029+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:51.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:51.029+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:51.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:51.029+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:51.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:50.818+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:51.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:50.818+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:51.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:50.818+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:51.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:50.819+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:51.686 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:51.194+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:51.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:51.194+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:51.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:51.194+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:51.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:51.194+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:52.178 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:51.827+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:52.178 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:51.827+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:52.179 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:51.827+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:52.179 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:51.827+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:52.179 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:52.065+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:52.179 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:52.065+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:52.179 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:52.065+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:52.180 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:52.065+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:52.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:52.181+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:52.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:52.181+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:52.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:52.181+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:52.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:52.181+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:53.186 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:52.818+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:53.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:52.818+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:53.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:52.818+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:53.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:52.818+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:53.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:53.045+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:53.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:53.045+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:53.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:53.045+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:53.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:53.045+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:53.636 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:20:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:20:53.328+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 28475 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:20:53.637 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:53.209+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:53.637 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:53.209+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:53.637 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:53.209+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:53.637 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:53.210+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:53.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:53.829+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:53.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:53.829+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:53.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:53.829+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:53.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:53.830+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:54.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:54.079+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:54.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:54.079+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:54.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:54.079+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:54.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:54.080+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:54.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:54.253+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:54.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:54.253+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:54.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:54.253+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:54.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:54.253+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:55.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:54.877+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:55.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:54.877+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:55.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:54.877+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:55.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:54.878+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:55.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:55.092+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:55.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:55.092+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:55.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:55.092+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:55.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:55.092+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:55.686 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:55.214+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:55.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:55.214+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:55.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:55.214+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:55.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:55.214+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:56.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:55.878+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:56.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:55.878+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:56.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:55.878+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:56.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:55.878+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:56.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:56.105+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:56.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:56.105+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:56.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:56.105+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:56.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:56.106+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:56.686 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:56.239+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:56.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:56.239+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:56.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:56.239+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:56.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:56.239+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:57.106 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:56.847+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:57.106 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:56.847+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:57.107 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:56.847+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:57.107 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:56.847+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:57.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:57.109+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:57.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:57.109+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:57.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:57.109+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:57.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:57.109+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:57.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:57.198+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:57.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:57.198+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:57.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:57.198+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:57.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:57.198+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:58.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:58.086+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:58.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:58.086+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:58.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:58.086+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:58.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:58.087+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:58.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:57.896+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:58.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:57.896+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:58.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:57.896+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:58.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:57.896+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:58.454 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:20:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:20:58.333+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 28564 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:20:58.454 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:58.194+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:58.455 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:58.194+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:58.455 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:58.194+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:58.455 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:58.194+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:59.097 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:58.849+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:59.097 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:58.849+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:59.097 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:58.849+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:20:59.097 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:58.849+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:20:59.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:59.100+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:59.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:59.100+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:59.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:59.100+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:20:59.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:20:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:20:59.100+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:20:59.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:59.204+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:59.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:59.204+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:59.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:59.204+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:20:59.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:20:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:20:59.204+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:00.111 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:59.858+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:00.111 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:59.858+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:00.111 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:59.858+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:00.111 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:20:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:20:59.858+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:00.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:00.114+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:00.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:00.114+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:00.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:00.114+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:00.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:00.114+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:00.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:00.199+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:00.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:00.199+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:00.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:00.199+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:00.439 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:00.199+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:01.127 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:00.825+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:01.127 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:00.825+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:01.127 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:00.825+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:01.127 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:00.826+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:01.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:01.130+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:01.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:01.130+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:01.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:01.130+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:01.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:01.130+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:01.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:01.187+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:01.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:01.187+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:01.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:01.187+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:01.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:01.187+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:02.094 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:01.847+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:02.094 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:01.847+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:02.094 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:01.847+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:02.094 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:01.847+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:02.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:02.096+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:02.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:02.096+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:02.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:02.096+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:02.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:02.097+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:02.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:02.186+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:02.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:02.186+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:02.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:02.186+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:02.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:02.187+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:03.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:03.079+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:03.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:03.079+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:03.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:03.079+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:03.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:03.079+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:03.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:02.894+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:03.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:02.894+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:03.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:02.894+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:03.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:02.894+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:03.637 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:21:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:21:03.338+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 28657 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:21:03.637 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:03.222+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:03.638 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:03.222+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:03.638 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:03.222+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:03.638 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:03.223+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:04.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:04.110+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:04.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:04.110+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:04.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:04.110+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:04.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:04.111+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:04.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:03.912+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:04.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:03.912+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:04.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:03.912+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:04.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:03.912+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:04.686 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:04.200+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:04.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:04.200+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:04.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:04.200+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:04.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:04.200+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:05.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:05.117+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:05.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:05.117+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:05.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:05.117+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:05.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:05.118+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:05.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:05.180+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:05.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:05.180+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:05.439 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:05.180+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:05.439 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:05.180+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:05.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:04.957+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:05.440 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:04.957+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:05.440 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:04.957+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:05.440 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:04.957+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:06.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:06.166+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:06.436 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:06.166+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:06.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:06.166+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:06.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:06.166+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:06.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:06.004+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:06.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:06.004+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:06.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:06.004+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:06.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:06.004+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:06.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:06.230+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:06.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:06.230+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:06.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:06.230+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:06.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:06.230+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:07.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:07.159+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:07.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:07.159+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:07.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:07.159+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:07.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:07.159+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:07.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:07.048+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:07.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:07.048+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:07.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:07.048+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:07.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:07.048+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:07.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:07.204+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:07.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:07.204+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:07.439 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:07.204+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:07.439 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:07.204+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:08.340 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:08.184+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:08.340 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:08.184+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:08.340 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:08.184+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:08.340 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:08.184+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:08.341 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:08.065+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:08.341 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:08.065+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:08.341 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:08.065+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:08.341 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:08.065+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:08.342 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:08.194+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:08.342 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:08.194+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:08.342 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:08.194+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:08.342 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:08.195+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:08.686 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:21:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:21:08.342+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 28746 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:21:09.182 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:09.100+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:09.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:09.100+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:09.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:09.100+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:09.183 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:09.100+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:09.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:09.149+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:09.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:09.149+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:09.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:09.149+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:09.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:09.149+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:09.436 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:09.186+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:09.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:09.186+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:09.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:09.186+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:09.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:09.186+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:10.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:10.095+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:10.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:10.095+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:10.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:10.095+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:10.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:10.095+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:10.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:10.193+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:10.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:10.193+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:10.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:10.193+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:10.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:10.193+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:10.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:10.203+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:10.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:10.203+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:10.439 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:10.203+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:10.439 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:10.203+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:11.436 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:11.128+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:11.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:11.128+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:11.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:11.128+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:11.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:11.128+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:11.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:11.233+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:11.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:11.233+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:11.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:11.233+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:11.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:11.233+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:11.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:11.156+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:11.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:11.156+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:11.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:11.156+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:11.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:11.156+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:12.436 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:12.169+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:12.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:12.169+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:12.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:12.169+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:12.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:12.169+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:12.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:12.186+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:12.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:12.186+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:12.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:12.186+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:12.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:12.186+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:12.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:12.171+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:12.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:12.171+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:12.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:12.171+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:12.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:12.171+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:13.368 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:21:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:21:13.347+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 28838 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:21:13.368 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:13.172+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:13.368 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:13.172+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:13.368 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:13.172+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:13.368 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:13.172+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:13.369 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:13.143+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:13.369 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:13.143+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:13.369 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:13.143+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:13.369 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:13.143+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:13.369 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:13.119+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:13.370 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:13.119+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:13.370 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:13.119+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:13.370 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:13.120+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:14.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:14.137+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:14.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:14.137+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:14.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:14.137+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:14.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:14.137+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:14.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:14.147+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:14.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:14.147+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:14.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:14.147+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:14.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:14.147+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:14.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:14.153+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:14.439 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:14.153+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:14.439 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:14.153+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:14.439 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:14.153+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:15.160 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:15.112+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:15.160 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:15.112+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:15.160 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:15.112+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:15.160 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:15.113+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:15.436 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:15.163+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:15.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:15.163+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:15.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:15.163+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:15.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:15.163+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:15.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:15.181+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:15.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:15.181+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:15.437 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:15.181+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:15.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:15.181+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:16.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:16.156+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:16.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:16.156+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:16.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:16.156+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:16.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:16.156+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:16.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:16.089+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:16.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:16.089+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:16.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:16.089+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:16.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:16.089+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:16.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:16.215+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:16.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:16.215+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:16.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:16.215+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:16.439 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:16.216+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:17.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:17.178+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:17.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:17.178+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:17.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:17.178+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:17.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:17.178+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:17.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:17.252+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:17.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:17.252+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:17.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:17.252+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:17.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:17.252+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:17.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:17.102+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:17.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:17.102+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:17.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:17.102+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:17.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:17.102+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:18.350 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:18.216+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:18.350 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:18.216+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:18.351 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:18.216+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:18.351 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:18.216+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:18.351 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:18.076+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:18.351 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:18.076+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:18.351 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:18.076+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:18.352 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:18.076+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:18.352 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:18.131+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:18.352 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:18.131+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:18.352 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:18.131+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:18.352 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:18.131+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:18.686 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:21:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:21:18.352+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 28926 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:21:19.436 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:19.087+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:19.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:19.087+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:19.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:19.087+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:19.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:19.087+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:19.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:19.044+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:19.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:19.044+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:19.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:19.044+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:19.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:19.045+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:19.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:19.174+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:19.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:19.174+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:19.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:19.174+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:19.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:19.174+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:20.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:20.055+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:20.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:20.055+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:20.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:20.055+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:20.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:20.055+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:20.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:20.013+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:20.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:20.013+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:20.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:20.013+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:20.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:20.013+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:20.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:20.172+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:20.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:20.172+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:20.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:20.172+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:20.439 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:20.173+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:21.436 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:21.018+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:21.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:21.018+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:21.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:21.018+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:21.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:21.018+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:21.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:20.969+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:21.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:20.969+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:21.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:20.969+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:21.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:20.969+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:21.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:21.131+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:21.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:21.131+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:21.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:21.131+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:21.439 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:21.131+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:22.338 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:21.955+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:22.338 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:21.955+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:22.338 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:21.955+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:22.339 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:21.955+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:22.339 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:22.093+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:22.339 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:22.093+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:22.339 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:22.093+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:22.339 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:22.093+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:22.340 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:22.028+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:22.340 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:22.028+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:22.340 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:22.028+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:22.340 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:22.028+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:23.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:22.914+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:23.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:22.914+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:23.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:22.914+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:23.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:22.914+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:23.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:23.057+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:23.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:23.057+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:23.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:23.057+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:23.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:23.057+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:23.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:23.029+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:23.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:23.029+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:23.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:23.029+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:23.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:23.029+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:23.638 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:21:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:21:23.356+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 29019 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:21:23.937 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:23.889+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:23.937 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:23.889+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:23.937 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:23.889+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:23.937 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:23.889+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:24.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:24.009+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:24.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:24.009+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:24.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:24.009+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:24.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:24.009+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:24.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:24.041+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:24.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:24.041+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:24.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:24.041+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:24.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:24.041+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:25.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:25.048+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:25.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:25.048+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:25.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:25.048+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:25.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:25.048+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:25.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:25.022+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:25.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:25.022+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:25.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:25.022+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:25.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:25.023+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:25.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:24.914+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:25.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:24.914+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:25.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:24.914+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:25.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:24.914+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:26.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:25.935+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:26.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:25.935+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:26.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:25.935+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:26.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:25.935+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:26.187 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:26.004+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:26.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:26.004+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:26.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:26.004+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:26.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:26.004+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:26.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:26.021+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:26.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:26.021+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:26.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:26.021+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:26.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:26.021+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:27.342 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:27.016+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:27.342 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:27.016+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:27.343 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:27.016+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:27.343 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:27.016+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:27.343 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:26.977+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:27.343 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:26.977+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:27.343 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:26.977+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:27.344 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:26.978+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:27.344 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:26.979+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:27.344 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:26.980+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:27.344 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:26.980+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:27.344 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:26.980+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:28.014 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:27.946+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:28.014 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:27.946+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:28.015 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:27.946+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:28.015 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:27.947+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:28.015 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:27.994+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:28.015 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:27.994+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:28.015 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:27.994+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:28.015 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:27.995+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:28.359 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:28.016+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:28.359 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:28.016+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:28.359 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:28.016+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:28.359 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:28.016+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:28.686 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:21:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:21:28.361+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 29108 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:21:29.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:28.945+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:29.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:28.945+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:29.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:28.945+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:29.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:28.946+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:29.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:29.011+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:29.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:29.011+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:29.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:29.011+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:29.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:29.012+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:29.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:29.034+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:29.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:29.034+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:29.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:29.034+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:29.439 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:29.034+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:30.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:29.906+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:30.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:29.906+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:30.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:29.906+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:30.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:29.906+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:30.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:30.052+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:30.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:30.052+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:30.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:30.052+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:30.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:30.053+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:30.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:30.011+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:30.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:30.011+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:30.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:30.011+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:30.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:30.011+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:31.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:30.865+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:31.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:30.865+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:31.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:30.865+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:31.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:30.865+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:31.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:30.985+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:31.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:30.986+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:31.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:30.986+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:31.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:30.986+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:31.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:31.083+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:31.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:31.083+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:31.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:31.083+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:31.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:31.083+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:32.186 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:32.049+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:32.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:32.049+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:32.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:32.049+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:32.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:32.050+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:32.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:31.903+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:32.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:31.903+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:32.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:31.903+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:32.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:31.904+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:32.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:31.941+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:32.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:31.941+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:32.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:31.941+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:32.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:31.941+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:33.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:33.080+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:33.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:33.080+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:33.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:33.080+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:33.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:33.080+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:33.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:32.857+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:33.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:32.857+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:33.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:32.857+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:33.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:32.857+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:33.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:32.906+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:33.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:32.906+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:33.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:32.906+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:33.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:32.906+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:33.638 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:21:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:21:33.366+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 29200 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:21:33.921 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:33.848+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:33.921 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:33.848+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:33.922 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:33.848+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:33.922 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:33.848+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:34.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:34.049+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:34.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:34.049+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:34.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:34.049+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:34.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:34.049+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:34.187 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:33.924+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:34.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:33.924+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:34.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:33.924+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:34.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:33.924+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:35.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:34.833+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:35.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:34.833+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:35.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:34.833+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:35.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:34.833+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:35.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:35.062+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:35.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:35.062+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:35.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:35.062+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:35.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:35.063+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:35.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:34.899+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:35.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:34.899+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:35.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:34.899+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:35.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:34.899+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:36.186 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:35.826+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:36.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:35.826+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:36.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:35.826+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:36.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:35.826+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:36.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:36.048+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:36.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:36.048+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:36.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:36.048+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:36.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:36.048+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:36.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:35.891+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:36.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:35.891+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:36.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:35.891+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:36.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:35.891+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:37.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:37.007+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:37.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:37.007+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:37.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:37.007+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:37.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:37.008+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:37.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:36.829+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:37.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:36.829+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:37.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:36.829+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:37.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:36.829+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:37.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:36.853+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:37.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:36.853+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:37.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:36.853+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:37.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:36.853+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:38.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:38.036+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:38.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:38.036+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:38.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:38.036+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:38.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:38.037+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:38.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:37.826+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:38.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:37.826+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:38.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:37.826+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:38.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:37.826+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:38.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:37.820+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:38.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:37.820+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:38.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:37.820+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:38.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:37.820+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:38.490 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:21:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:21:38.369+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 29289 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:21:39.058 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:38.801+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:39.059 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:38.801+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:39.059 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:38.801+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:39.059 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:38.801+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:39.059 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:38.822+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:39.059 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:38.822+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:39.059 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:38.822+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:39.060 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:38.822+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:39.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:39.061+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:39.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:39.061+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:39.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:39.061+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:39.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:39.061+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:40.014 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:39.836+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:40.014 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:39.836+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:40.014 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:39.836+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:40.015 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:39.836+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:40.015 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:39.815+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:40.015 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:39.815+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:40.015 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:39.815+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:40.015 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:39.816+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:40.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:40.085+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:40.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:40.085+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:40.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:40.085+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:40.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:40.086+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:41.118 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:40.848+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:41.119 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:40.848+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:41.119 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:40.848+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:41.119 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:40.849+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:41.119 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:40.859+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:41.119 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:40.859+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:41.119 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:40.859+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:41.120 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:40.859+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:41.436 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:41.121+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:41.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:41.121+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:41.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:41.121+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:41.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:41.121+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:42.146 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:41.825+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:42.147 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:41.825+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:42.147 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:41.825+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:42.147 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:41.825+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:42.147 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:41.897+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:42.147 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:41.897+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:42.147 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:41.897+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:42.148 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:41.897+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:42.436 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:42.148+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:42.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:42.148+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:42.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:42.148+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:42.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:42.149+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:43.147 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:42.797+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:43.147 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:42.797+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:43.147 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:42.797+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:43.148 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:42.798+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:43.148 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:42.930+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:43.148 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:42.930+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:43.148 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:42.930+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:43.148 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:42.930+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:43.437 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:21:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:21:43.374+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 29382 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:21:43.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:43.149+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:43.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:43.149+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:43.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:43.149+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:43.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:43.150+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:44.139 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:43.767+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:44.139 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:43.767+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:44.139 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:43.767+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:44.140 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:43.767+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:44.140 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:43.897+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:44.140 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:43.897+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:44.140 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:43.897+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:44.140 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:43.897+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:44.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:44.142+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:44.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:44.142+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:44.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:44.142+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:44.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:44.142+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:45.132 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:44.760+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:45.132 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:44.760+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:45.132 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:44.760+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:45.133 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:44.760+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:45.133 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:44.879+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:45.133 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:44.879+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:45.133 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:44.879+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:45.133 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:44.879+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:45.436 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:45.134+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:45.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:45.134+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:45.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:45.134+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:45.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:45.135+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:45.893 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:45.746+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:45.894 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:45.746+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:45.894 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:45.746+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:45.894 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:45.746+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:46.144 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:45.896+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:46.144 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:45.896+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:46.144 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:45.896+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:46.145 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:45.896+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:46.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:46.147+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:46.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:46.147+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:46.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:46.147+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:46.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:46.147+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:47.144 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:46.758+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:47.144 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:46.758+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:47.145 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:46.758+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:47.145 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:46.758+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:47.145 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:46.906+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:47.145 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:46.906+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:47.145 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:46.906+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:47.145 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:46.906+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:47.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:47.147+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:47.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:47.147+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:47.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:47.147+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:47.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:47.147+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:48.159 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:47.794+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:48.159 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:47.794+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:48.159 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:47.794+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:48.159 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:47.794+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:48.160 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:47.920+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:48.160 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:47.920+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:48.160 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:47.920+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:48.160 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:47.920+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:48.437 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:21:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:21:48.379+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 29471 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:21:48.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:48.161+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:48.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:48.161+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:48.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:48.161+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:48.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:48.161+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:49.130 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:48.881+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:49.130 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:48.881+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:49.131 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:48.881+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:49.131 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:48.881+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:49.131 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:48.795+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:49.131 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:48.795+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:49.131 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:48.795+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:49.131 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:48.796+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:49.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:49.132+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:49.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:49.132+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:49.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:49.132+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:49.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:49.132+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:50.119 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:49.916+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:50.120 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:49.916+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:50.120 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:49.916+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:50.120 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:49.917+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:50.121 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:49.784+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:50.121 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:49.784+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:50.121 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:49.784+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:50.121 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:49.784+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:50.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:50.121+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:50.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:50.121+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:50.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:50.121+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:50.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:50.122+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:51.013 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:50.882+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:51.014 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:50.882+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:51.014 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:50.882+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:51.014 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:50.883+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:51.014 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:50.761+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:51.014 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:50.761+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:51.015 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:50.761+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:51.015 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:50.761+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:51.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:51.111+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:51.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:51.111+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:51.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:51.111+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:51.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:51.111+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:52.013 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:51.725+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:52.013 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:51.725+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:52.013 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:51.725+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:52.013 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:51.726+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:52.013 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:51.846+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:52.014 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:51.846+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:52.014 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:51.846+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:52.014 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:51.846+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:52.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:52.109+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:52.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:52.109+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:52.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:52.109+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:52.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:52.109+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:53.099 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:52.823+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:53.099 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:52.823+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:53.099 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:52.823+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:53.099 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:52.823+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:53.100 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:52.760+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:53.100 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:52.760+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:53.100 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:52.760+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:53.100 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:52.760+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:53.382 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:53.102+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:53.382 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:53.102+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:53.382 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:53.102+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:53.382 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:53.102+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:53.640 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:21:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:21:53.384+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 29563 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:21:53.937 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:53.789+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:53.937 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:53.789+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:53.937 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:53.789+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:53.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:53.789+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:53.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:53.833+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:53.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:53.833+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:53.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:53.833+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:53.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:53.834+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:54.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:54.100+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:54.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:54.100+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:54.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:54.100+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:54.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:54.100+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:55.136 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:54.811+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:55.136 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:54.811+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:55.136 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:54.811+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:55.137 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:54.811+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:55.137 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:54.818+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:55.137 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:54.818+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:55.137 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:54.818+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:55.137 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:54.818+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:55.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:55.138+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:55.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:55.138+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:55.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:55.138+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:55.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:55.139+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:56.132 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:55.855+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:56.132 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:55.855+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:56.132 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:55.855+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:56.132 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:55.855+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:56.133 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:55.866+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:56.133 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:55.866+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:56.133 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:55.866+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:56.133 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:55.866+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:56.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:56.134+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:56.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:56.134+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:56.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:56.134+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:56.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:56.134+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:57.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:56.820+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:57.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:56.820+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:57.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:56.820+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:57.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:56.820+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:57.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:56.880+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:57.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:56.880+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:57.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:56.880+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:57.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:56.880+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:57.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:57.184+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:57.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:57.184+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:57.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:57.184+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:57.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:57.184+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:58.132 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:57.845+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:58.133 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:57.845+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:58.133 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:57.845+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:58.133 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:57.846+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:58.133 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:57.899+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:58.133 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:57.899+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:58.134 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:57.899+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:58.134 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:57.899+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:58.387 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:58.135+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:58.387 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:58.135+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:58.387 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:58.135+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:58.387 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:58.135+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:58.687 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:21:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:21:58.389+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 29652 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:21:59.170 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:58.884+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:59.171 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:58.884+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:59.171 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:58.884+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:21:59.171 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:58.884+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:21:59.171 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:58.836+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:59.171 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:58.836+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:59.171 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:58.836+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:21:59.171 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:58.836+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:21:59.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:59.172+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:59.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:59.172+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:59.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:59.172+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:21:59.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:21:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:21:59.173+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:00.162 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:59.905+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:00.162 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:59.905+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:00.162 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:59.905+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:00.162 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:21:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:21:59.905+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:00.163 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:59.872+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:00.163 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:59.872+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:00.163 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:59.872+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:00.163 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:21:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:21:59.872+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:00.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:00.164+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:00.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:00.164+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:00.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:00.164+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:00.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:00.164+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:01.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:00.925+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:01.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:00.925+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:01.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:00.925+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:01.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:00.926+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:01.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:00.836+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:01.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:00.836+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:01.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:00.836+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:01.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:00.836+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:01.687 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:01.201+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:01.687 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:01.201+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:01.687 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:01.201+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:01.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:01.201+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:02.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:01.885+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:02.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:01.885+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:02.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:01.885+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:02.184 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:01.885+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:02.184 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:01.791+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:02.184 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:01.791+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:02.184 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:01.791+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:02.185 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:01.791+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:02.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:02.185+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:02.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:02.185+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:02.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:02.185+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:02.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:02.186+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:03.139 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:02.888+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:03.139 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:02.888+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:03.140 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:02.888+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:03.140 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:02.889+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:03.140 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:02.803+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:03.140 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:02.803+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:03.140 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:02.803+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:03.140 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:02.803+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:03.392 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:03.141+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:03.392 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:03.141+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:03.392 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:03.141+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:03.392 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:03.142+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:03.687 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:22:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:22:03.394+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 29745 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:22:04.153 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:03.927+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:04.153 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:03.927+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:04.154 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:03.927+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:04.154 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:03.928+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:04.154 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:03.770+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:04.154 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:03.770+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:04.154 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:03.770+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:04.154 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:03.771+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:04.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:04.156+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:04.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:04.156+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:04.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:04.156+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:04.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:04.156+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:05.141 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:04.738+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:05.141 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:04.738+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:05.141 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:04.738+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:05.141 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:04.738+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:05.142 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:04.961+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:05.142 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:04.961+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:05.142 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:04.961+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:05.142 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:04.961+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:05.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:05.143+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:05.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:05.143+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:05.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:05.143+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:05.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:05.144+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:05.965 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:05.711+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:05.965 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:05.711+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:05.965 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:05.711+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:05.965 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:05.711+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:06.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:06.156+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:06.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:06.156+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:06.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:06.156+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:06.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:06.156+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:06.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:05.968+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:06.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:05.968+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:06.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:05.968+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:06.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:05.968+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:06.970 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:06.718+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:06.970 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:06.718+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:06.971 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:06.718+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:06.971 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:06.718+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:07.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:07.142+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:07.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:07.142+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:07.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:07.142+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:07.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:07.142+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:07.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:06.973+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:07.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:06.973+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:07.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:06.973+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:07.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:06.973+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:08.004 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:07.736+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:08.005 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:07.736+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:08.005 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:07.736+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:08.005 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:07.737+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:08.397 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:08.007+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:08.397 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:08.007+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:08.398 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:08.007+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:08.398 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:08.007+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:08.398 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:08.141+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:08.398 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:08.141+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:08.398 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:08.141+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:08.398 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:08.141+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:08.689 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:22:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:22:08.399+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 29834 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:22:09.117 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:08.987+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:09.118 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:08.987+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:09.118 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:08.987+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:09.118 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:08.987+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:09.118 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:08.776+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:09.118 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:08.776+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:09.119 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:08.776+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:09.119 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:08.777+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:09.425 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:09.121+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:09.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:09.121+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:09.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:09.121+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:09.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:09.121+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:10.098 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:09.973+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:10.098 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:09.973+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:10.098 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:09.973+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:10.098 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:09.973+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:10.098 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:09.797+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:10.099 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:09.797+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:10.099 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:09.797+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:10.099 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:09.797+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:10.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:10.099+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:10.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:10.099+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:10.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:10.099+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:10.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:10.099+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:11.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:11.070+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:11.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:11.070+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:11.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:11.070+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:11.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:11.070+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:11.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:10.971+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:11.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:10.971+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:11.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:10.971+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:11.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:10.971+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:11.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:10.836+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:11.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:10.836+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:11.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:10.836+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:11.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:10.836+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:12.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:12.055+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:12.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:12.055+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:12.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:12.055+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:12.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:12.055+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:12.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:12.012+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:12.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:12.012+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:12.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:12.012+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:12.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:12.012+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:12.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:11.868+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:12.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:11.868+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:12.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:11.868+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:12.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:11.868+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:13.187 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:12.855+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:13.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:12.855+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:13.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:12.855+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:13.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:12.856+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:13.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:13.036+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:13.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:13.036+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:13.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:13.036+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:13.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:13.037+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:13.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:13.060+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:13.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:13.060+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:13.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:13.060+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:13.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:13.061+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:13.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:22:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:22:13.404+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 29925 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:22:14.106 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:13.820+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:14.107 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:13.820+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:14.107 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:13.820+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:14.107 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:13.820+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:14.107 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:14.028+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:14.107 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:14.028+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:14.107 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:14.028+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:14.108 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:14.029+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:14.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:14.108+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:14.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:14.108+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:14.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:14.108+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:14.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:14.109+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:15.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:14.980+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:15.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:14.980+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:15.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:14.980+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:15.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:14.980+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:15.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:14.855+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:15.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:14.855+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:15.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:14.855+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:15.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:14.856+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:15.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:15.099+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:15.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:15.099+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:15.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:15.099+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:15.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:15.100+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:16.106 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:15.845+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:16.106 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:15.845+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:16.107 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:15.845+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:16.107 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:15.845+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:16.107 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:15.989+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:16.107 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:15.989+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:16.107 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:15.989+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:16.107 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:15.989+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:16.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:16.108+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:16.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:16.108+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:16.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:16.108+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:16.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:16.109+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:17.148 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:16.880+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:17.149 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:16.880+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:17.149 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:16.880+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:17.149 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:16.881+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:17.149 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:16.947+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:17.150 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:16.947+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:17.150 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:16.947+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:17.150 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:16.948+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:17.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:17.151+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:17.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:17.151+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:17.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:17.151+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:17.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:17.151+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:18.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:17.904+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:18.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:17.904+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:18.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:17.904+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:18.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:17.905+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:18.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:17.897+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:18.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:17.897+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:18.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:17.897+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:18.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:17.898+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:18.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:18.109+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:18.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:18.109+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:18.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:18.109+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:18.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:18.109+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:18.687 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:22:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:22:18.409+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 30015 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:22:19.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:18.898+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:19.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:18.898+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:19.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:18.898+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:19.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:18.899+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:19.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:18.919+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:19.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:18.919+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:19.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:18.919+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:19.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:18.919+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:19.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:19.098+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:19.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:19.098+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:19.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:19.098+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:19.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:19.098+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:20.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:19.909+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:20.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:19.909+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:20.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:19.909+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:20.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:19.910+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:20.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:19.874+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:20.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:19.874+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:20.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:19.874+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:20.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:19.875+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:20.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:20.105+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:20.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:20.105+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:20.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:20.105+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:20.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:20.106+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:21.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:20.901+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:21.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:20.901+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:21.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:20.901+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:21.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:20.902+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:21.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:20.878+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:21.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:20.878+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:21.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:20.878+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:21.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:20.878+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:21.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:21.098+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:21.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:21.098+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:21.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:21.098+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:21.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:21.098+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:22.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:21.867+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:22.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:21.867+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:22.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:21.867+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:22.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:21.868+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:22.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:21.909+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:22.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:21.909+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:22.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:21.909+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:22.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:21.909+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:22.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:22.079+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:22.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:22.079+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:22.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:22.079+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:22.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:22.080+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:23.187 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:22.926+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:23.187 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:22.926+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:23.187 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:22.926+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:23.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:22.926+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:23.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:23.076+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:23.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:23.076+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:23.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:23.076+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:23.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:23.076+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:23.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:22.826+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:23.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:22.826+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:23.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:22.826+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:23.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:22.827+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:23.687 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:22:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:22:23.414+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 30106 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:22:24.058 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:23.806+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:24.058 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:23.806+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:24.058 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:23.806+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:24.059 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:23.806+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:24.059 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:23.966+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:24.059 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:23.966+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:24.059 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:23.966+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:24.059 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:23.966+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:24.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:24.060+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:24.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:24.060+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:24.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:24.060+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:24.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:24.060+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:24.934 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:24.794+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:24.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:24.794+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:24.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:24.794+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:24.935 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:24.795+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:25.187 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:24.936+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:25.187 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:24.936+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:25.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:24.936+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:25.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:24.937+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:25.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:25.055+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:25.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:25.055+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:25.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:25.055+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:25.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:25.056+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:26.100 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:25.835+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:26.100 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:25.835+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:26.100 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:25.835+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:26.100 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:25.835+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:26.101 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:25.889+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:26.101 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:25.889+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:26.101 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:25.889+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:26.101 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:25.889+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:26.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:26.102+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:26.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:26.102+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:26.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:26.102+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:26.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:26.102+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:27.143 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:26.874+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:27.143 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:26.874+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:27.143 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:26.874+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:27.144 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:26.875+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:27.144 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:26.863+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:27.144 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:26.863+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:27.144 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:26.863+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:27.144 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:26.864+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:27.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:27.145+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:27.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:27.145+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:27.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:27.145+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:27.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:27.146+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:28.123 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:27.872+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:28.123 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:27.872+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:28.123 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:27.872+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:28.123 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:27.872+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:28.124 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:27.822+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:28.124 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:27.822+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:28.124 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:27.822+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:28.124 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:27.822+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:28.417 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:28.122+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:28.417 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:28.122+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:28.418 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:28.122+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:28.418 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:28.122+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:28.687 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:22:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:22:28.419+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 30195 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:22:29.105 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:28.913+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:29.106 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:28.913+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:29.106 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:28.913+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:29.106 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:28.913+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:29.106 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:28.824+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:29.106 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:28.824+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:29.106 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:28.824+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:29.107 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:28.825+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:29.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:29.107+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:29.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:29.107+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:29.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:29.107+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:29.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:29.108+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:30.155 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:29.871+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:30.155 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:29.871+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:30.155 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:29.871+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:30.155 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:29.871+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:30.156 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:29.910+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:30.156 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:29.910+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:30.156 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:29.910+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:30.156 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:29.910+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:30.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:30.157+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:30.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:30.157+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:30.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:30.157+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:30.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:30.157+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:31.180 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:30.892+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:31.181 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:30.892+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:31.181 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:30.892+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:31.181 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:30.892+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:31.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:30.855+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:31.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:30.855+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:31.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:30.855+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:31.182 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:30.855+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:31.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:31.182+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:31.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:31.182+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:31.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:31.182+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:31.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:31.183+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:31.937 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:31.853+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:31.937 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:31.853+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:31.937 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:31.853+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:31.937 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:31.854+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:32.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:31.939+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:32.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:31.939+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:32.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:31.939+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:32.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:31.939+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:32.687 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:32.230+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:32.687 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:32.230+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:32.688 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:32.230+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:32.688 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:32.230+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:33.187 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:32.902+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:33.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:32.902+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:33.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:32.902+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:33.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:32.902+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:33.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:32.982+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:33.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:32.982+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:33.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:32.982+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:33.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:32.982+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:33.544 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:22:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:22:33.424+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 30285 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:22:33.545 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:33.275+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:33.545 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:33.275+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:33.545 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:33.275+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:33.545 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:33.275+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:34.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:33.997+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:34.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:33.997+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:34.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:33.997+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:34.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:33.997+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:34.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:33.896+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:34.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:33.896+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:34.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:33.896+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:34.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:33.896+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:34.636 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:34.261+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:34.636 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:34.261+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:34.636 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:34.261+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:00.710420+0000 (oldest deadline 2024-04-06T03:12:20.710420+0000) 2024-04-06T03:22:34.636 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:34.262+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:34.937 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:34.854+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:34.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:34.854+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:34.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:34.854+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:34.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:34.855+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:35.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:35.246+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:35.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:35.026+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:35.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:35.026+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:35.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:35.026+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:35.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:35.027+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:36.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:36.055+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:36.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:36.055+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:36.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:36.055+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:36.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:36.055+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:36.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:35.832+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:36.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:35.832+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:36.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:35.832+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:36.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:35.832+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:36.687 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:36.284+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:37.078 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:36.821+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:37.078 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:36.821+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:37.078 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:36.821+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:37.078 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:36.821+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:37.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:37.260+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:37.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:37.079+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:37.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:37.079+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:37.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:37.079+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:37.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:37.079+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:38.071 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:37.797+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:38.071 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:37.797+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:38.071 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:37.797+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:38.072 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:37.797+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:38.427 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:38.253+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:38.427 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:38.073+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:38.427 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:38.073+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:38.427 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:38.073+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:38.427 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:38.073+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:38.687 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:22:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:22:38.428+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 30374 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:22:39.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:39.034+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:39.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:39.034+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:39.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:39.034+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:39.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:39.035+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:39.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:38.804+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:39.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:38.804+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:39.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:38.804+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:07.716355+0000 (oldest deadline 2024-04-06T03:12:27.716355+0000) 2024-04-06T03:22:39.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:38.805+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:39.678 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:39.246+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:39.937 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:39.850+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:40.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:40.229+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:40.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:40.003+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:40.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:40.003+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:40.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:40.003+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:40.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:40.004+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:41.187 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:40.858+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:41.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:40.979+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:41.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:40.979+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:41.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:40.979+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:12:06.593557+0000 (oldest deadline 2024-04-06T03:12:26.593557+0000) 2024-04-06T03:22:41.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:40.979+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:41.687 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:41.226+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:42.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:41.938+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:42.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:41.810+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:42.687 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:42.247+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:43.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:42.958+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:43.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:42.810+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:43.555 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:22:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:22:43.433+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 30465 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:22:43.555 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:43.215+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:44.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:43.936+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:44.187 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:43.817+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:44.687 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:44.253+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:44.977 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:44.818+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:45.266 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:44.979+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:45.687 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:45.268+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:46.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:45.968+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:46.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:45.836+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:46.687 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:46.259+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:47.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:46.971+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:47.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:46.791+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:47.687 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:47.307+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:48.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:47.921+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:48.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:47.791+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:48.560 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:22:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:22:48.438+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 30555 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:22:48.561 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:48.285+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:48.914 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:48.795+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:49.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:48.917+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:49.687 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:49.239+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:50.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:49.924+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:50.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:49.812+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:50.687 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:50.203+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:51.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:50.896+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:51.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:50.854+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:51.687 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:51.204+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:52.162 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:51.875+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:52.163 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:51.806+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:52.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:52.165+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:53.170 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:52.854+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:53.171 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:52.769+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:53.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:53.182+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:53.723 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:22:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:22:53.444+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 30645 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:22:54.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:53.854+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:54.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:53.725+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:54.687 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:54.215+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:55.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:54.860+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:55.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:54.721+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:55.684 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:55.196+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:55.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:55.895+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:55.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:55.686+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:56.687 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:56.201+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:57.169 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:56.940+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:57.169 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:56.700+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:57.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:57.171+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:58.181 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:57.915+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:58.181 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:57.735+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:58.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:58.183+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:22:58.866 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:22:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:22:58.449+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 30729 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:22:58.866 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:58.694+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:59.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:58.868+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:22:59.687 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:59.219+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:22:59.688 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:59.219+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:22:59.688 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:59.219+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:22:59.688 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:22:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:22:59.219+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:00.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:22:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:22:59.836+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:00.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:22:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:22:59.721+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:00.686 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:00.260+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:00.686 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:00.260+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:00.686 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:00.260+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:00.687 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:00.260+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:00.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:00.886+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:00.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:00.687+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:00.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:00.687+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:00.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:00.687+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:00.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:00.687+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:01.654 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:01.222+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:01.655 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:01.222+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:01.655 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:01.222+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:01.655 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:01.222+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:01.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:01.898+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:01.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:01.656+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:01.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:01.656+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:01.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:01.656+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:01.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:01.656+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:02.386 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:02.211+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:02.386 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:02.211+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:02.386 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:02.211+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:02.387 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:02.211+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:02.687 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:02.631+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:02.688 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:02.631+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:02.688 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:02.631+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:02.688 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:02.631+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:03.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:02.936+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:03.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:02.936+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:03.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:02.936+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:03.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:02.936+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:03.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:03.163+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:03.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:03.163+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:03.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:03.163+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:03.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:03.163+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:03.937 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:23:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:23:03.454+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 30820 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:23:03.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:03.674+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:03.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:03.674+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:03.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:03.674+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:03.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:03.674+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:04.437 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:03.954+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:04.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:03.954+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:04.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:03.954+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:04.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:03.954+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:04.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:04.184+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:04.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:04.184+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:04.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:04.184+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:04.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:04.185+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:04.937 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:04.656+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:04.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:04.656+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:04.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:04.656+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:04.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:04.657+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:05.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:04.978+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:05.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:04.978+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:05.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:04.978+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:05.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:04.978+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:05.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:05.224+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:05.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:05.224+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:05.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:05.224+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:05.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:05.224+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:05.937 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:05.609+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:05.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:05.609+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:05.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:05.609+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:05.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:05.609+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:06.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:06.176+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:06.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:06.176+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:06.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:06.176+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:06.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:06.176+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:06.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:05.939+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:06.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:05.939+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:06.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:05.939+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:06.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:05.940+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:06.937 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:06.632+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:06.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:06.632+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:06.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:06.632+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:06.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:06.633+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:07.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:06.960+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:07.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:06.960+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:07.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:06.960+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:07.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:06.961+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:07.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:07.165+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:07.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:07.165+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:07.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:07.165+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:07.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:07.165+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:07.937 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:07.592+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:07.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:07.592+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:07.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:07.592+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:07.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:07.592+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:08.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:08.137+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:08.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:08.137+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:08.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:08.137+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:08.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:08.137+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:08.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:07.945+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:08.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:07.945+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:08.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:07.945+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:08.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:07.946+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:08.894 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:23:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:23:08.460+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 30910 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:23:08.895 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:08.564+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:08.895 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:08.564+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:08.895 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:08.564+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:08.895 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:08.564+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:09.149 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:08.896+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:09.150 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:08.896+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:09.150 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:08.896+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:09.150 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:08.897+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:09.437 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:09.151+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:09.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:09.151+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:09.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:09.151+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:09.438 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:09.152+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:09.891 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:09.551+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:09.891 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:09.551+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:09.891 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:09.551+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:09.892 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:09.551+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:10.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:10.134+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:10.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:10.134+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:10.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:10.134+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:10.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:10.134+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:10.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:09.893+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:10.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:09.893+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:10.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:09.893+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:10.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:09.893+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:10.883 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:10.543+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:10.883 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:10.543+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:10.883 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:10.543+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:10.883 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:10.543+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:11.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:11.133+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:11.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:11.133+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:11.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:11.133+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:11.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:11.134+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:11.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:10.884+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:11.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:10.885+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:11.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:10.885+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:11.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:10.885+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:11.888 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:11.545+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:11.888 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:11.545+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:11.888 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:11.545+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:11.888 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:11.545+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:12.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:12.097+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:12.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:12.097+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:12.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:12.097+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:12.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:12.098+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:12.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:11.889+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:12.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:11.889+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:12.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:11.889+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:12.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:11.890+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:12.871 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:12.507+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:12.871 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:12.507+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:12.871 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:12.507+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:12.872 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:12.508+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:13.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:13.097+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:13.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:13.097+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:13.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:13.097+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:13.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:13.098+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:13.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:12.873+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:13.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:12.873+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:13.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:12.873+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:13.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:12.873+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:13.847 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:23:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:23:13.465+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 31000 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:23:13.847 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:13.473+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:13.847 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:13.474+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:13.847 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:13.474+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:13.848 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:13.474+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:14.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:14.069+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:14.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:14.069+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:14.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:14.069+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:14.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:14.069+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:14.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:13.848+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:14.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:13.848+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:14.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:13.848+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:14.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:13.849+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:14.870 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:14.493+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:14.871 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:14.493+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:14.871 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:14.493+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:14.871 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:14.493+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:15.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:14.872+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:15.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:14.872+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:15.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:14.872+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:15.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:14.872+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:15.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:15.054+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:15.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:15.054+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:15.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:15.054+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:15.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:15.055+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:15.905 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:15.541+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:15.905 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:15.541+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:15.905 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:15.541+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:15.905 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:15.541+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:16.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:15.906+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:16.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:15.906+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:16.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:15.906+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:16.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:15.907+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:16.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:16.015+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:16.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:16.015+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:16.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:16.015+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:16.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:16.015+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:16.924 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:16.577+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:16.924 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:16.577+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:16.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:16.577+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:16.925 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:16.577+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:17.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:16.926+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:17.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:16.926+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:17.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:16.926+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:17.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:16.926+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:17.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:16.999+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:17.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:16.999+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:17.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:16.999+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:17.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:16.999+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:17.903 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:17.617+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:17.904 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:17.617+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:17.904 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:17.617+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:17.904 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:17.617+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:18.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:17.949+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:18.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:17.949+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:18.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:17.949+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:18.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:17.950+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:18.190 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:17.905+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:18.190 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:17.905+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:18.190 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:17.905+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:18.190 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:17.905+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:18.872 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:23:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:23:18.471+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 31090 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:23:18.872 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:18.636+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:18.872 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:18.636+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:18.873 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:18.636+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:18.873 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:18.637+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:19.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:18.873+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:19.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:18.873+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:19.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:18.873+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:19.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:18.873+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:19.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:18.928+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:19.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:18.928+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:19.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:18.928+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:19.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:18.928+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:19.862 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:19.614+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:19.863 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:19.614+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:19.863 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:19.614+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:19.863 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:19.614+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:20.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:19.887+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:20.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:19.887+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:20.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:19.887+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:20.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:19.887+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:20.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:19.864+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:20.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:19.864+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:20.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:19.864+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:20.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:19.864+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:20.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:20.864+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:20.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:20.864+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:20.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:20.864+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:20.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:20.864+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:20.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:20.840+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:20.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:20.840+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:20.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:20.840+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:20.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:20.840+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:20.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:20.637+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:20.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:20.637+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:20.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:20.637+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:20.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:20.637+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:21.864 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:21.838+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:21.864 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:21.838+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:21.864 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:21.838+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:21.865 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:21.838+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:21.865 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:21.608+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:21.865 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:21.608+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:21.865 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:21.608+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:21.865 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:21.608+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:22.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:21.866+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:22.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:21.866+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:22.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:21.866+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:22.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:21.866+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:22.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:22.862+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:22.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:22.862+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:22.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:22.862+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:22.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:22.863+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:22.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:22.842+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:22.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:22.842+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:22.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:22.842+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:22.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:22.842+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:22.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:22.635+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:22.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:22.635+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:22.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:22.635+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:22.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:22.635+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:23.813 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:23:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:23:23.477+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 31181 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:23:23.814 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:23.678+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:23.814 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:23.678+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:23.814 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:23.678+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:23.814 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:23.679+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:24.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:23.842+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:24.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:23.842+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:24.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:23.842+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:24.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:23.842+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:24.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:23.815+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:24.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:23.815+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:24.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:23.815+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:24.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:23.815+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:24.937 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:24.839+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:24.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:24.839+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:24.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:24.839+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:24.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:24.839+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:24.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:24.639+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:24.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:24.639+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:24.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:24.639+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:24.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:24.639+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:24.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:24.805+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:24.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:24.805+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:24.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:24.805+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:24.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:24.805+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:25.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:25.800+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:25.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:25.800+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:25.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:25.800+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:25.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:25.800+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:25.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:25.633+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:25.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:25.633+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:25.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:25.633+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:25.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:25.634+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:25.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:25.788+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:25.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:25.788+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:25.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:25.788+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:25.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:25.788+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:26.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:26.770+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:26.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:26.770+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:26.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:26.770+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:26.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:26.770+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:26.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:26.811+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:26.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:26.811+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:26.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:26.811+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:26.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:26.812+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:26.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:26.587+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:26.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:26.587+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:26.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:26.587+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:26.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:26.587+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:27.938 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:27.781+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:27.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:27.781+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:27.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:27.781+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:27.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:27.781+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:27.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:27.743+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:27.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:27.743+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:27.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:27.743+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:27.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:27.743+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:27.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:27.619+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:27.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:27.619+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:27.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:27.619+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:27.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:27.619+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:28.827 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:23:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:23:28.482+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 31271 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:23:28.827 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:28.577+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:28.827 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:28.577+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:28.827 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:28.577+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:28.828 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:28.577+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:28.828 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:28.719+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:28.828 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:28.719+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:28.828 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:28.719+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:28.828 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:28.719+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:29.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:28.829+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:29.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:28.829+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:29.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:28.829+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:29.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:28.829+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:29.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:29.608+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:29.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:29.608+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:29.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:29.608+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:29.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:29.608+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:29.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:29.817+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:29.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:29.817+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:29.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:29.817+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:29.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:29.818+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:29.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:29.671+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:29.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:29.671+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:29.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:29.671+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:29.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:29.671+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:30.841 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:30.689+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:30.842 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:30.689+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:30.842 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:30.689+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:30.842 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:30.689+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:30.842 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:30.571+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:30.843 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:30.571+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:30.843 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:30.571+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:30.843 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:30.571+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:31.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:30.843+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:31.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:30.843+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:31.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:30.843+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:31.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:30.843+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:31.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:31.650+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:31.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:31.650+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:31.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:31.650+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:31.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:31.650+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:31.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:31.795+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:31.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:31.795+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:31.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:31.795+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:31.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:31.795+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:31.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:31.548+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:31.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:31.548+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:31.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:31.548+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:31.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:31.548+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:32.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:32.689+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:32.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:32.689+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:32.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:32.689+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:32.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:32.689+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:32.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:32.588+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:32.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:32.588+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:32.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:32.588+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:32.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:32.588+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:32.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:32.829+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:32.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:32.829+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:32.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:32.829+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:32.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:32.829+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:33.845 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:23:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:23:33.488+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 31361 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:23:33.845 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:33.584+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:33.845 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:33.584+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:33.846 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:33.584+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:33.846 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:33.584+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:33.846 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:33.720+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:33.846 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:33.720+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:33.846 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:33.720+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:33.846 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:33.720+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:34.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:33.847+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:34.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:33.847+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:34.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:33.847+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:34.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:33.847+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:34.861 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:34.672+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:34.861 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:34.672+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:34.862 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:34.672+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:34.862 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:34.672+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:34.862 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:34.607+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:34.862 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:34.607+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:34.862 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:34.607+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:34.863 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:34.607+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:35.144 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:34.863+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:35.145 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:34.863+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:35.145 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:34.863+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:35.145 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:34.863+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:35.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:35.648+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:35.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:35.648+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:35.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:35.648+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:35.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:35.649+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:35.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:35.632+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:35.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:35.632+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:35.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:35.632+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:35.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:35.632+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:35.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:35.866+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:35.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:35.866+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:35.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:35.866+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:35.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:35.866+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:36.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:36.607+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:36.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:36.607+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:36.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:36.607+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:36.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:36.607+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:36.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:36.624+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:36.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:36.624+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:36.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:36.624+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:36.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:36.625+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:36.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:36.819+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:36.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:36.819+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:36.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:36.819+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:36.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:36.819+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:37.868 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:37.573+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:37.868 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:37.573+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:37.868 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:37.573+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:37.869 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:37.573+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:37.869 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:37.622+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:37.869 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:37.622+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:37.870 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:37.622+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:37.870 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:37.622+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:38.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:37.869+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:38.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:37.869+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:38.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:37.869+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:38.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:37.869+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:38.904 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:23:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:23:38.493+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 31451 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:23:38.904 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:38.665+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:38.905 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:38.665+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:38.905 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:38.665+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:38.905 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:38.665+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:38.905 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:38.588+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:38.905 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:38.588+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:38.905 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:38.588+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:38.906 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:38.588+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:39.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:38.906+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:39.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:38.906+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:39.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:38.906+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:39.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:38.906+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:39.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:39.642+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:39.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:39.642+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:39.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:39.642+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:39.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:39.642+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:39.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:39.583+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:39.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:39.583+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:39.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:39.583+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:39.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:39.584+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:40.218 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:39.942+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:40.226 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:39.942+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:40.226 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:39.942+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:40.226 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:39.942+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:40.898 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:40.537+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:40.899 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:40.537+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:40.899 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:40.537+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:40.899 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:40.537+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:40.899 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:40.650+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:40.899 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:40.650+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:40.899 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:40.650+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:40.900 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:40.650+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:41.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:40.900+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:41.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:40.900+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:41.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:40.900+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:41.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:40.900+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:41.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:41.648+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:41.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:41.648+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:41.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:41.648+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:41.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:41.648+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:41.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:41.554+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:41.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:41.554+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:41.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:41.554+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:41.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:41.554+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:42.322 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:41.946+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:42.323 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:41.946+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:42.323 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:41.946+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:42.323 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:41.946+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:42.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:42.586+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:42.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:42.586+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:42.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:42.586+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:42.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:42.586+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:42.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:42.612+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:42.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:42.612+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:42.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:42.612+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:42.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:42.612+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:43.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:42.978+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:43.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:42.978+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:43.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:42.978+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:43.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:42.978+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:43.936 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:23:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:23:43.499+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 31542 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:23:43.936 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:43.600+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:43.936 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:43.600+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:43.937 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:43.600+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:43.937 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:43.600+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:43.937 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:43.622+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:43.937 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:43.622+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:43.937 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:43.622+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:43.937 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:43.622+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:44.187 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:43.938+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:44.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:43.938+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:44.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:43.938+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:44.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:43.938+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:44.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:44.635+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:44.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:44.635+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:44.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:44.635+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:44.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:44.635+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:44.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:44.668+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:44.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:44.668+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:44.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:44.668+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:44.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:44.668+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:45.229 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:44.952+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:45.229 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:44.952+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:45.230 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:44.952+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:45.230 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:44.952+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:45.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:45.651+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:45.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:45.651+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:45.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:45.651+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:45.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:45.651+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:45.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:45.621+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:45.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:45.621+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:45.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:45.621+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:45.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:45.622+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:46.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:45.944+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:46.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:45.944+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:46.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:45.944+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:46.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:45.945+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:46.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:46.652+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:46.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:46.652+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:46.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:46.652+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:46.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:46.653+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:46.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:46.617+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:46.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:46.617+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:46.928 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:46.617+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:46.928 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:46.617+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:47.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:46.928+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:47.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:46.928+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:47.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:46.928+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:47.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:46.928+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:47.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:47.586+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:47.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:47.586+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:47.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:47.586+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:47.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:47.586+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:47.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:47.631+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:47.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:47.631+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:47.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:47.631+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:47.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:47.631+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:48.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:47.965+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:48.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:47.965+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:48.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:47.965+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:48.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:47.965+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:48.938 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:23:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:23:48.504+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 31632 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:23:48.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:48.550+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:48.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:48.550+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:48.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:48.550+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:48.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:48.550+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:48.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:48.623+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:48.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:48.623+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:48.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:48.623+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:48.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:48.623+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:49.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:48.953+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:49.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:48.953+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:49.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:48.953+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:49.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:48.953+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:49.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:49.513+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:49.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:49.513+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:49.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:49.513+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:49.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:49.513+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:49.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:49.635+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:49.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:49.635+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:49.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:49.635+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:49.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:49.635+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:50.271 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:50.001+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:50.272 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:50.001+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:50.272 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:50.001+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:50.272 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:50.001+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:50.683 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:50.522+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:50.683 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:50.522+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:50.684 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:50.522+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:50.684 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:50.522+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:50.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:50.685+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:50.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:50.685+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:50.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:50.685+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:50.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:50.685+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:51.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:50.972+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:51.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:50.972+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:51.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:50.972+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:51.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:50.972+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:51.926 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:51.656+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:51.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:51.656+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:51.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:51.656+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:51.927 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:51.656+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:51.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:51.540+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:51.927 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:51.540+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:51.928 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:51.540+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:51.928 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:51.540+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:52.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:51.928+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:52.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:51.928+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:52.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:51.928+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:52.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:51.928+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:52.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:52.657+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:52.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:52.657+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:52.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:52.657+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:52.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:52.657+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:52.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:52.576+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:52.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:52.576+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:52.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:52.576+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:52.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:52.577+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:53.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:52.939+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:53.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:52.939+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:53.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:52.939+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:53.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:52.940+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:53.935 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:23:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:23:53.510+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 31722 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:23:53.936 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:53.685+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:53.936 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:53.685+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:53.936 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:53.685+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:53.936 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:53.685+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:53.936 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:53.565+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:53.936 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:53.565+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:53.936 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:53.565+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:53.937 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:53.565+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:54.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:53.936+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:54.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:53.936+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:54.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:53.936+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:54.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:53.937+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:54.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:54.718+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:54.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:54.718+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:54.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:54.718+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:54.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:54.718+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:54.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:54.611+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:54.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:54.611+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:54.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:54.611+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:54.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:54.611+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:55.314 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:54.970+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:55.314 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:54.970+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:55.314 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:54.970+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:55.314 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:54.971+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:55.564 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:55.565+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:55.564 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:55.565+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:55.564 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:55.565+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:55.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:55.566+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:55.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:55.686+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:55.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:55.686+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:55.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:55.686+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:55.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:55.686+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:56.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:55.992+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:56.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:55.992+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:56.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:55.992+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:56.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:55.992+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:56.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:56.691+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:56.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:56.691+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:56.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:56.691+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:56.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:56.692+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:56.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:56.594+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:56.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:56.594+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:56.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:56.594+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:56.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:56.595+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:57.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:57.025+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:57.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:57.025+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:57.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:57.025+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:57.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:57.026+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:57.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:57.696+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:57.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:57.696+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:57.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:57.696+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:57.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:57.697+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:57.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:57.550+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:57.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:57.550+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:57.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:57.550+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:57.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:57.550+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:58.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:58.043+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:58.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:58.043+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:58.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:58.043+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:58.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:58.044+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:58.938 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:23:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:23:58.515+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 31812 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:23:58.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:58.740+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:58.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:58.740+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:58.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:58.740+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:58.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:58.740+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:58.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:58.569+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:58.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:58.569+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:58.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:58.569+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:58.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:58.569+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:23:59.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:58.996+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:59.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:58.996+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:59.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:58.996+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:23:59.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:58.997+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:59.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:59.741+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:59.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:59.741+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:59.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:59.741+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:23:59.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:23:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:23:59.741+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:23:59.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:59.521+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:59.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:59.521+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:59.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:59.521+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:23:59.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:23:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:23:59.522+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:00.220 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:59.957+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:00.220 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:59.957+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:00.220 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:59.957+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:00.220 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:23:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:23:59.958+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:00.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:00.562+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:00.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:00.562+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:00.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:00.562+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:00.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:00.563+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:00.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:00.766+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:00.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:00.766+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:00.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:00.766+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:00.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:00.766+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:01.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:00.990+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:01.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:00.990+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:01.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:00.990+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:01.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:00.990+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:01.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:01.572+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:01.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:01.573+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:01.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:01.573+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:01.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:01.573+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:01.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:01.780+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:01.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:01.780+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:01.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:01.780+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:01.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:01.780+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:02.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:02.002+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:02.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:02.002+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:02.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:02.002+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:02.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:02.002+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:02.804 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:02.546+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:02.804 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:02.546+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:02.805 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:02.546+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:02.805 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:02.546+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:03.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:03.016+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:03.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:03.016+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:03.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:03.016+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:03.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:03.016+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:03.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:02.806+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:03.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:02.806+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:03.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:02.806+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:03.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:02.806+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:03.809 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:24:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:24:03.521+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 31903 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:24:03.810 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:03.550+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:03.810 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:03.550+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:03.810 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:03.550+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:03.810 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:03.550+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:04.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:04.057+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:04.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:04.057+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:04.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:04.057+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:04.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:04.057+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:04.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:03.811+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:04.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:03.811+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:04.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:03.811+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:04.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:03.811+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:04.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:04.570+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:04.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:04.570+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:04.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:04.570+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:04.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:04.570+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:04.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:04.805+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:04.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:04.805+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:04.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:04.805+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:04.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:04.806+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:05.399 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:05.035+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:05.399 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:05.035+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:05.400 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:05.035+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:05.400 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:05.035+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:05.688 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:05.568+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:05.688 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:05.568+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:05.688 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:05.568+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:05.688 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:05.569+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:06.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:06.023+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:06.190 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:06.023+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:06.190 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:06.023+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:06.190 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:06.023+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:06.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:05.777+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:06.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:05.777+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:06.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:05.777+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:06.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:05.777+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:06.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:06.589+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:06.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:06.589+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:06.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:06.589+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:06.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:06.589+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:06.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:06.823+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:06.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:06.823+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:06.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:06.823+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:06.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:06.823+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:07.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:06.985+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:07.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:06.985+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:07.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:06.985+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:07.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:06.985+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:07.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:07.631+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:07.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:07.631+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:07.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:07.631+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:07.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:07.631+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:07.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:07.822+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:07.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:07.822+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:07.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:07.822+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:07.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:07.822+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:08.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:08.032+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:08.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:08.032+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:08.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:08.032+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:08.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:08.033+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:08.850 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:24:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:24:08.527+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 31993 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:24:08.851 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:08.636+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:08.851 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:08.636+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:08.851 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:08.636+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:08.851 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:08.636+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:09.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:09.025+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:09.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:09.025+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:09.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:09.025+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:09.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:09.026+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:09.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:08.849+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:09.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:08.849+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:09.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:08.849+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:09.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:08.850+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:09.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:09.671+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:09.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:09.671+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:09.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:09.671+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:09.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:09.671+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:09.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:09.893+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:09.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:09.893+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:09.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:09.893+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:09.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:09.893+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:10.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:10.037+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:10.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:10.037+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:10.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:10.037+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:10.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:10.038+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:11.019 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:10.705+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:11.019 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:10.705+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:11.020 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:10.705+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:11.020 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:10.705+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:11.020 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:10.878+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:11.020 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:10.878+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:11.020 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:10.878+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:11.020 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:10.879+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:11.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:11.020+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:11.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:11.020+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:11.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:11.020+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:11.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:11.021+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:12.014 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:11.925+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:12.015 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:11.925+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:12.015 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:11.925+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:12.015 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:11.925+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:12.016 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:11.739+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:12.016 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:11.739+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:12.016 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:11.739+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:12.016 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:11.739+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:12.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:12.048+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:12.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:12.048+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:12.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:12.048+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:12.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:12.048+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:13.034 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:12.710+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:13.034 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:12.710+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:13.034 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:12.710+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:13.035 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:12.710+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:13.035 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:12.889+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:13.035 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:12.889+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:13.035 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:12.889+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:13.035 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:12.889+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:13.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:13.035+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:13.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:13.035+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:13.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:13.035+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:13.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:13.036+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:13.929 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:24:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:24:13.533+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 32083 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:24:13.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:13.679+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:13.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:13.679+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:13.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:13.679+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:13.930 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:13.679+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:14.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:14.071+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:14.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:14.071+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:14.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:14.071+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:14.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:14.071+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:14.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:13.931+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:14.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:13.931+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:14.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:13.931+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:14.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:13.931+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:15.104 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:14.903+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:15.105 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:14.903+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:15.105 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:14.903+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:15.105 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:14.903+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:15.105 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:14.719+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:15.105 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:14.719+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:15.105 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:14.719+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:15.106 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:14.719+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:15.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:15.105+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:15.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:15.105+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:15.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:15.105+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:15.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:15.106+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:15.879 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:15.723+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:15.879 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:15.723+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:15.880 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:15.723+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:15.880 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:15.723+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:16.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:16.087+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:16.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:16.087+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:16.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:16.087+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:16.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:16.087+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:16.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:15.880+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:16.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:15.880+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:16.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:15.880+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:16.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:15.881+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:16.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:16.878+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:16.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:16.878+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:16.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:16.878+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:16.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:16.878+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:16.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:16.676+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:16.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:16.676+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:16.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:16.676+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:16.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:16.676+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:17.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:17.121+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:17.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:17.121+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:17.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:17.121+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:17.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:17.121+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:18.134 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:17.694+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:18.134 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:17.694+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:18.134 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:17.694+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:18.134 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:17.695+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:18.135 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:17.847+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:18.135 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:17.847+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:18.135 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:17.847+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:18.135 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:17.847+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:18.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:18.135+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:18.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:18.135+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:18.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:18.135+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:18.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:18.135+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:18.864 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:24:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:24:18.538+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 32173 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:24:18.864 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:18.711+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:18.864 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:18.711+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:18.865 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:18.711+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:18.865 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:18.711+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:19.151 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:18.865+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:19.152 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:18.865+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:19.152 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:18.865+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:19.152 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:18.865+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:19.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:19.152+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:19.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:19.152+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:19.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:19.152+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:19.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:19.153+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:20.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:19.698+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:20.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:19.698+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:20.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:19.698+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:20.187 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:19.698+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:20.187 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:19.853+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:20.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:19.853+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:20.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:19.853+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:20.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:19.853+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:20.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:20.188+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:20.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:20.188+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:20.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:20.188+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:20.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:20.188+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:20.825 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:20.695+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:20.825 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:20.695+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:20.825 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:20.695+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:20.826 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:20.696+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:21.141 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:20.826+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:21.141 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:20.826+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:21.141 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:20.826+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:21.142 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:20.826+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:21.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:21.143+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:21.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:21.143+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:21.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:21.143+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:21.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:21.143+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:21.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:21.667+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:21.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:21.667+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:21.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:21.667+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:21.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:21.668+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:21.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:21.871+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:21.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:21.871+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:21.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:21.871+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:21.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:21.871+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:22.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:22.180+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:22.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:22.180+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:22.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:22.180+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:22.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:22.180+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:22.870 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:22.629+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:22.871 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:22.629+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:22.871 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:22.629+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:22.871 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:22.629+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:23.144 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:22.871+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:23.144 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:22.871+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:23.144 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:22.871+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:23.144 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:22.872+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:23.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:23.146+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:23.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:23.146+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:23.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:23.146+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:23.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:23.146+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:23.901 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:24:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:24:23.544+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 32264 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:24:23.902 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:23.647+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:23.902 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:23.647+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:23.902 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:23.647+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:23.902 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:23.647+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:24.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:24.114+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:24.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:24.114+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:24.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:24.114+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:24.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:24.114+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:24.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:23.903+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:24.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:23.903+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:24.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:23.903+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:24.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:23.903+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:24.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:24.676+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:24.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:24.676+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:24.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:24.676+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:24.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:24.676+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:24.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:24.894+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:24.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:24.894+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:24.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:24.894+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:24.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:24.895+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:25.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:25.152+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:25.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:25.152+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:25.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:25.152+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:25.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:25.152+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:25.888 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:25.635+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:25.889 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:25.635+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:25.889 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:25.635+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:25.889 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:25.635+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:26.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:25.890+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:26.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:25.890+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:26.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:25.890+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:26.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:25.890+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:26.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:26.128+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:26.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:26.128+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:26.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:26.128+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:26.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:26.128+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:26.923 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:26.651+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:26.923 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:26.651+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:26.924 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:26.651+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:26.924 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:26.651+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:27.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:27.134+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:27.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:27.134+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:27.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:27.134+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:27.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:27.134+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:27.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:26.925+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:27.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:26.925+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:27.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:26.925+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:27.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:26.925+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:27.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:27.683+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:27.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:27.683+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:27.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:27.683+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:27.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:27.683+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:28.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:27.959+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:28.438 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:27.959+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:28.439 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:27.959+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:28.439 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:27.959+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:28.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:28.108+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:28.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:28.108+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:28.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:28.108+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:28.440 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:28.109+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:28.917 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:24:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:24:28.549+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 32354 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:24:28.917 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:28.695+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:28.917 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:28.695+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:28.918 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:28.695+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:28.918 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:28.695+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:29.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:28.919+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:29.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:28.919+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:29.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:28.919+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:29.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:28.919+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:29.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:29.110+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:29.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:29.110+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:29.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:29.110+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:29.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:29.110+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:29.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:29.658+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:29.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:29.658+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:29.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:29.658+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:29.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:29.658+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:29.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:29.880+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:29.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:29.880+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:29.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:29.880+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:29.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:29.880+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:30.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:30.126+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:30.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:30.126+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:30.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:30.126+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:30.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:30.126+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:30.878 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:30.701+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:30.878 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:30.701+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:30.878 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:30.701+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:30.878 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:30.701+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:31.152 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:30.879+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:31.152 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:30.879+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:31.153 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:30.879+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:31.153 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:30.879+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:31.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:31.154+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:31.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:31.154+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:31.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:31.154+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:31.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:31.154+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:31.920 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:31.721+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:31.920 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:31.721+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:31.920 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:31.721+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:31.920 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:31.721+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:32.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:32.134+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:32.188 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:32.134+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:32.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:32.134+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:32.189 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:32.134+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:32.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:31.922+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:32.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:31.922+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:32.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:31.922+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:32.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:31.922+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:33.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:32.753+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:33.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:32.753+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:33.182 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:32.753+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:33.183 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:32.753+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:33.183 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:32.926+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:33.183 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:32.926+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:33.183 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:32.926+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:33.183 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:32.926+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:33.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:33.184+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:33.438 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:33.184+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:33.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:33.184+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:33.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:33.184+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:33.886 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:24:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:24:33.555+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 32444 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:24:33.887 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:33.737+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:33.887 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:33.737+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:33.887 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:33.737+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:33.888 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:33.737+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:34.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:33.887+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:34.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:33.887+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:34.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:33.887+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:34.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:33.887+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:34.547 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:34.213+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:34.547 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:34.213+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:34.547 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:34.213+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:34.548 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:34.214+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:34.882 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:34.718+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:34.883 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:34.718+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:34.883 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:34.718+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:34.883 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:34.718+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:35.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:34.884+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:35.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:34.884+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:35.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:34.884+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:35.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:34.884+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:35.653 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:35.203+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:35.653 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:35.203+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:35.653 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:35.203+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:35.653 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:35.203+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:35.911 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:35.708+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:35.911 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:35.708+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:35.911 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:35.708+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:35.911 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:35.708+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:36.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:35.913+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:36.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:35.913+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:36.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:35.913+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:36.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:35.913+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:36.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:36.193+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:36.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:36.193+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:36.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:36.193+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:36.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:36.194+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:37.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:36.692+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:37.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:36.692+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:37.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:36.692+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:37.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:36.692+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:37.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:36.884+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:37.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:36.884+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:37.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:36.884+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:37.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:36.885+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:37.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:37.197+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:37.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:37.197+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:37.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:37.197+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:37.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:37.197+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:38.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:37.922+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:38.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:37.922+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:38.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:37.922+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:38.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:37.922+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:38.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:37.733+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:38.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:37.733+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:38.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:37.733+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:38.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:37.733+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:38.560 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:38.199+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:38.561 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:38.199+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:38.561 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:38.199+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:38.561 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:38.199+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:38.914 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:24:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:24:38.560+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 32531 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:24:38.914 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:38.766+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:38.914 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:38.766+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:38.914 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:38.766+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:38.915 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:38.766+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:39.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:38.915+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:39.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:38.915+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:39.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:38.915+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:39.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:38.915+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:39.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:39.220+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:39.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:39.220+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:39.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:39.220+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:39.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:39.220+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:40.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:39.772+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:40.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:39.772+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:40.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:39.772+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:40.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:39.772+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:40.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:39.887+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:40.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:39.887+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:40.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:39.887+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:40.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:39.887+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:40.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:40.223+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:40.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:40.223+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:40.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:40.223+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:40.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:40.223+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:41.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:40.820+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:41.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:40.820+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:41.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:40.820+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:41.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:40.821+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:41.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:40.856+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:41.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:40.856+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:41.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:40.856+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:41.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:40.856+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:41.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:41.204+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:41.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:41.204+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:41.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:41.204+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:41.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:41.204+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:42.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:41.846+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:42.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:41.846+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:42.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:41.846+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:42.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:41.847+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:42.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:41.840+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:42.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:41.840+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:42.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:41.840+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:42.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:41.840+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:42.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:42.250+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:42.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:42.250+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:42.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:42.250+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:42.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:42.251+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:43.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:42.846+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:43.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:42.846+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:43.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:42.846+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:43.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:42.847+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:43.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:42.865+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:43.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:42.865+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:43.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:42.865+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:43.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:42.865+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:43.566 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:43.284+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:43.566 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:43.285+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:43.567 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:43.285+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:43.567 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:43.285+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:43.824 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:24:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:24:43.567+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 32613 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:24:44.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:43.827+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:44.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:43.827+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:44.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:43.827+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:44.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:43.827+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:44.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:43.825+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:44.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:43.825+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:44.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:43.825+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:44.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:43.825+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:44.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:44.330+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:44.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:44.330+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:44.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:44.330+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:44.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:44.331+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:45.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:44.807+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:45.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:44.807+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:45.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:44.807+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:45.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:44.808+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:45.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:44.826+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:45.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:44.826+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:45.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:44.826+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:45.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:44.826+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:45.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:45.349+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:45.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:45.349+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:45.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:45.349+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:45.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:45.349+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:46.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:45.836+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:46.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:45.836+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:46.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:45.836+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:46.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:45.836+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:46.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:45.826+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:46.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:45.826+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:46.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:45.826+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:46.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:45.827+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:46.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:46.363+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:46.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:46.363+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:46.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:46.363+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:46.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:46.363+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:47.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:46.854+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:47.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:46.854+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:47.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:46.854+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:47.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:46.854+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:47.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:46.845+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:47.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:46.845+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:47.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:46.846+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:47.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:46.846+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:47.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:47.396+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:47.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:47.396+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:47.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:47.396+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:47.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:47.397+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:48.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:47.863+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:48.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:47.863+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:48.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:47.863+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:48.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:47.864+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:48.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:47.840+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:48.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:47.840+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:48.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:47.840+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:48.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:47.840+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:48.689 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:24:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:24:48.573+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 32694 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:24:48.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:48.368+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:48.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:48.368+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:48.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:48.368+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:48.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:48.368+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:49.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:48.821+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:49.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:48.821+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:49.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:48.821+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:49.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:48.822+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:49.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:48.880+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:49.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:48.880+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:49.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:48.880+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:49.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:48.880+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:49.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:49.362+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:49.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:49.362+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:49.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:49.362+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:49.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:49.362+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:50.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:49.804+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:50.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:49.804+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:50.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:49.804+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:50.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:49.804+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:50.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:49.871+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:50.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:49.871+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:50.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:49.871+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:50.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:49.871+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:50.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:50.379+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:50.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:50.379+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:50.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:50.379+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:50.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:50.379+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:51.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:50.800+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:51.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:50.800+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:51.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:50.800+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:51.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:50.800+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:51.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:50.864+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:51.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:50.864+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:51.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:50.864+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:51.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:50.864+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:51.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:51.428+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:51.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:51.428+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:51.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:51.428+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:51.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:51.429+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:52.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:51.826+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:52.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:51.826+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:52.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:51.826+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:52.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:51.826+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:52.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:51.850+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:52.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:51.850+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:52.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:51.850+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:52.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:51.850+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:52.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:52.383+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:52.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:52.383+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:52.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:52.383+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:52.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:52.383+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:53.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:52.781+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:53.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:52.781+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:53.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:52.781+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:53.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:52.781+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:53.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:52.842+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:53.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:52.842+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:53.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:52.842+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:53.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:52.842+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:53.654 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:24:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:24:53.579+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 32775 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:24:53.655 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:53.388+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:53.655 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:53.388+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:53.655 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:53.388+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:53.655 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:53.388+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:53.938 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:53.782+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:53.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:53.782+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:53.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:53.782+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:53.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:53.783+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:53.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:53.866+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:53.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:53.866+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:53.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:53.866+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:53.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:53.867+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:54.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:54.403+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:54.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:54.403+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:54.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:54.403+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:54.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:54.403+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:55.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:54.830+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:55.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:54.830+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:55.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:54.830+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:55.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:54.831+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:55.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:54.819+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:55.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:54.819+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:55.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:54.819+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:55.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:54.819+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:55.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:55.361+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:55.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:55.362+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:55.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:55.362+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:55.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:55.362+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:56.051 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:55.798+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:56.052 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:55.798+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:56.052 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:55.798+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:56.052 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:55.799+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:56.053 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:55.859+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:56.053 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:55.859+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:56.053 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:55.859+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:56.053 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:55.859+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:56.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:56.360+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:56.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:56.360+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:56.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:56.360+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:56.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:56.361+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:57.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:56.899+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:57.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:56.899+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:57.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:56.899+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:57.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:56.899+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:57.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:56.789+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:57.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:56.789+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:57.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:56.789+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:57.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:56.789+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:57.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:57.406+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:57.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:57.406+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:57.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:57.406+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:57.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:57.406+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:58.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:57.749+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:58.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:57.749+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:58.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:57.749+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:58.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:57.750+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:58.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:57.915+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:58.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:57.915+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:58.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:57.915+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:58.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:57.915+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:58.689 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:24:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:24:58.585+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 32856 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:24:58.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:58.438+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:58.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:58.438+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:58.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:58.438+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:58.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:58.439+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:59.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:58.732+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:59.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:58.732+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:59.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:58.732+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:24:59.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:58.732+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:24:59.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:58.939+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:59.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:58.939+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:59.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:58.939+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:24:59.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:58.939+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:24:59.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:59.396+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:59.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:59.396+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:59.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:59.396+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:24:59.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:24:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:24:59.397+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:00.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:59.771+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:00.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:59.771+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:00.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:59.771+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:00.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:24:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:24:59.771+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:00.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:59.947+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:00.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:59.947+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:00.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:59.947+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:00.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:24:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:24:59.947+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:00.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:00.412+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:00.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:00.412+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:00.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:00.412+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:00.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:00.412+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:01.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:00.780+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:01.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:00.780+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:01.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:00.780+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:01.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:00.780+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:01.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:00.904+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:01.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:00.904+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:01.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:00.904+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:01.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:00.904+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:01.777 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:01.447+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:01.778 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:01.447+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:01.778 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:01.447+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:01.778 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:01.447+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:02.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:01.778+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:02.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:01.778+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:02.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:01.778+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:02.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:01.779+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:02.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:01.869+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:02.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:01.869+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:02.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:01.869+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:02.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:01.870+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:02.771 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:02.478+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:02.771 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:02.478+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:02.771 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:02.478+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:02.771 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:02.479+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:03.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:02.771+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:03.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:02.771+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:03.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:02.771+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:03.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:02.772+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:03.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:02.908+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:03.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:02.908+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:03.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:02.908+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:03.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:02.908+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:03.778 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:25:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:25:03.591+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 32947 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:25:03.778 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:03.478+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:03.778 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:03.478+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:03.778 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:03.478+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:03.779 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:03.478+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:04.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:03.779+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:04.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:03.779+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:04.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:03.779+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:04.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:03.779+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:04.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:03.884+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:04.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:03.884+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:04.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:03.884+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:04.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:03.884+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:04.821 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:04.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:04.821 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:04.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:04.821 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:04.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:04.821 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:04.501+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:05.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:04.822+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:05.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:04.822+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:05.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:04.822+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:05.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:04.822+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:05.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:04.885+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:05.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:04.885+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:05.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:04.885+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:05.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:04.885+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:05.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:05.487+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:05.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:05.487+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:05.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:05.487+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:05.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:05.487+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:06.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:05.844+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:06.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:05.844+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:06.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:05.844+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:06.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:05.845+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:06.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:05.891+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:06.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:05.891+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:06.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:05.891+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:06.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:05.891+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:06.860 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:06.510+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:06.860 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:06.510+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:06.860 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:06.510+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:06.860 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:06.510+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:07.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:06.860+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:07.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:06.860+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:07.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:06.860+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:07.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:06.861+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:07.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:06.876+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:07.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:06.876+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:07.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:06.876+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:07.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:06.877+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:07.848 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:07.506+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:07.848 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:07.506+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:07.848 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:07.506+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:07.849 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:07.507+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:08.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:07.849+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:08.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:07.849+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:08.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:07.849+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:08.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:07.849+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:08.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:07.913+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:08.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:07.913+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:08.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:07.913+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:08.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:07.913+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:08.719 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:25:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:25:08.597+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 33037 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:25:08.719 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:08.461+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:08.719 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:08.461+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:08.720 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:08.461+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:08.720 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:08.461+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:09.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:08.845+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:09.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:08.845+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:09.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:08.845+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:09.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:08.845+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:09.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:08.904+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:09.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:08.904+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:09.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:08.904+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:09.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:08.904+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:09.824 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:09.486+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:09.825 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:09.486+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:09.825 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:09.486+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:09.825 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:09.487+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:10.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:09.825+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:10.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:09.825+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:10.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:09.825+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:10.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:09.825+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:10.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:09.890+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:10.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:09.890+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:10.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:09.890+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:10.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:09.891+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:10.793 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:10.492+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:10.793 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:10.492+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:10.793 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:10.492+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:10.793 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:10.493+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:11.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:10.793+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:11.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:10.793+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:11.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:10.793+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:11.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:10.794+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:11.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:10.932+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:11.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:10.932+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:11.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:10.932+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:11.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:10.932+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:11.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:11.535+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:11.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:11.535+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:11.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:11.535+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:11.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:11.535+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:12.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:11.816+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:12.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:11.816+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:12.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:11.816+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:12.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:11.816+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:12.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:11.939+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:12.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:11.939+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:12.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:11.939+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:12.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:11.939+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:12.778 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:12.514+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:12.778 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:12.514+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:12.778 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:12.514+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:12.778 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:12.514+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:13.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:12.778+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:13.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:12.778+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:13.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:12.778+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:13.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:12.779+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:13.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:12.939+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:13.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:12.939+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:13.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:12.939+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:13.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:12.939+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:13.938 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:25:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:25:13.603+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 33127 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:25:13.938 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:13.506+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:13.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:13.506+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:13.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:13.506+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:13.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:13.506+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:13.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:13.746+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:13.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:13.746+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:13.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:13.746+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:13.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:13.746+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:14.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:13.939+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:14.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:13.939+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:14.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:13.939+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:14.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:13.939+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:14.778 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:14.541+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:14.779 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:14.541+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:14.779 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:14.541+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:14.779 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:14.541+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:15.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:14.780+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:15.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:14.780+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:15.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:14.780+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:15.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:14.780+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:15.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:14.941+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:15.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:14.941+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:15.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:14.941+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:15.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:14.941+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:15.812 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:15.522+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:15.812 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:15.522+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:15.813 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:15.522+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:15.813 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:15.522+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:16.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:15.813+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:16.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:15.813+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:16.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:15.813+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:16.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:15.813+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:16.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:15.900+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:16.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:15.900+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:16.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:15.900+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:16.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:15.900+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:16.770 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:16.473+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:16.770 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:16.473+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:16.771 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:16.473+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:16.771 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:16.473+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:17.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:16.888+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:17.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:16.888+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:17.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:16.888+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:17.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:16.888+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:17.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:16.771+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:17.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:16.771+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:17.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:16.771+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:17.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:16.771+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:17.770 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:17.513+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:17.770 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:17.513+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:17.771 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:17.513+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:17.771 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:17.514+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:18.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:17.771+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:18.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:17.771+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:18.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:17.771+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:18.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:17.771+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:18.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:17.871+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:18.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:17.871+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:18.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:17.871+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:18.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:17.871+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:18.735 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:25:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:25:18.609+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 33217 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:25:18.736 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:18.470+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:18.736 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:18.470+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:18.736 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:18.470+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:18.737 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:18.470+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:19.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:18.736+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:19.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:18.736+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:19.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:18.736+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:19.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:18.736+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:19.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:18.833+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:19.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:18.833+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:19.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:18.833+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:19.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:18.834+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:19.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:19.432+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:19.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:19.432+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:19.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:19.432+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:19.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:19.432+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:20.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:19.751+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:20.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:19.751+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:20.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:19.751+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:20.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:19.751+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:20.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:19.847+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:20.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:19.847+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:20.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:19.847+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:20.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:19.847+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:20.627 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:20.418+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:20.627 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:20.418+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:20.628 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:20.418+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:20.628 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:20.418+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:20.895 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:20.709+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:20.895 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:20.709+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:20.895 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:20.709+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:20.895 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:20.709+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:21.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:20.895+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:21.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:20.895+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:21.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:20.895+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:21.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:20.896+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:21.736 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:21.468+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:21.736 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:21.468+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:21.736 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:21.468+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:21.737 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:21.468+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:22.014 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:21.913+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:22.014 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:21.913+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:22.014 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:21.913+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:22.014 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:21.913+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:22.014 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:21.737+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:22.015 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:21.737+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:22.015 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:21.737+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:22.015 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:21.737+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:22.758 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:22.464+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:22.758 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:22.464+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:22.758 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:22.464+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:22.758 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:22.464+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:23.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:22.759+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:23.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:22.759+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:23.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:22.759+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:23.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:22.759+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:23.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:22.949+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:23.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:22.949+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:23.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:22.949+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:23.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:22.949+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:23.740 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:25:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:25:23.615+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 33308 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:25:23.740 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:23.462+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:23.740 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:23.462+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:23.740 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:23.462+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:23.741 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:23.462+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:24.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:23.923+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:24.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:23.923+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:24.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:23.923+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:24.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:23.923+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:24.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:23.791+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:24.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:23.791+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:24.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:23.791+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:24.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:23.791+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:24.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:24.428+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:24.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:24.428+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:24.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:24.428+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:24.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:24.428+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:25.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:24.906+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:25.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:24.906+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:25.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:24.906+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:25.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:24.906+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:25.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:24.808+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:25.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:24.808+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:25.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:24.808+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:25.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:24.808+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:25.760 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:25.466+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:25.760 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:25.466+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:25.760 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:25.466+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:25.760 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:25.466+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:26.080 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:25.760+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:26.080 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:25.760+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:26.080 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:25.760+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:26.080 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:25.761+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:26.081 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:25.870+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:26.081 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:25.870+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:26.081 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:25.871+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:26.081 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:25.871+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:26.777 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:26.462+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:26.778 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:26.462+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:26.778 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:26.462+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:26.778 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:26.462+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:27.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:26.835+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:27.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:26.835+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:27.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:26.835+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:27.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:26.835+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:27.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:26.778+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:27.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:26.778+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:27.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:26.778+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:27.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:26.778+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:27.800 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:27.511+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:27.800 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:27.511+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:27.800 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:27.511+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:27.800 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:27.511+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:27.801 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:27.750+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:27.801 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:27.750+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:27.801 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:27.750+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:27.801 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:27.750+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:28.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:27.799+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:28.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:27.799+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:28.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:27.799+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:28.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:27.800+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:28.745 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:25:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:25:28.620+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 33398 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:25:28.745 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:28.477+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:28.745 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:28.477+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:28.745 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:28.477+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:28.746 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:28.477+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:28.746 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:28.702+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:28.746 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:28.702+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:28.746 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:28.702+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:28.746 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:28.702+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:29.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:28.797+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:29.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:28.797+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:29.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:28.797+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:29.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:28.797+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:29.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:29.437+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:29.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:29.437+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:29.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:29.437+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:29.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:29.437+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:30.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:29.722+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:30.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:29.722+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:30.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:29.722+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:30.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:29.722+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:30.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:29.825+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:30.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:29.825+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:30.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:29.825+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:30.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:29.825+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:30.731 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:30.731+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:30.731 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:30.731+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:30.732 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:30.731+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:30.732 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:30.732+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:30.732 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:30.482+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:30.732 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:30.482+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:30.732 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:30.482+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:30.732 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:30.482+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:31.122 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:30.798+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:31.122 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:30.798+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:31.122 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:30.798+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:31.122 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:30.798+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:31.728 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:31.472+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:31.728 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:31.472+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:31.729 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:31.472+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:31.729 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:31.472+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:32.014 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:31.729+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:32.014 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:31.729+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:32.014 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:31.729+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:32.014 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:31.730+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:32.015 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:31.838+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:32.015 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:31.838+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:32.015 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:31.838+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:32.015 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:31.838+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:32.740 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:32.475+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:32.740 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:32.475+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:32.740 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:32.475+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:32.741 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:32.475+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:33.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:32.740+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:33.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:32.740+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:33.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:32.740+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:33.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:32.741+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:33.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:32.835+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:33.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:32.835+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:33.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:32.835+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:33.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:32.835+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:33.784 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:25:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:25:33.626+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 33486 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:25:33.784 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:33.524+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:33.785 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:33.524+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:33.785 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:33.524+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:33.785 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:33.524+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:34.188 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:33.785+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:34.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:33.785+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:34.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:33.785+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:34.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:33.785+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:34.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:33.791+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:34.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:33.791+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:34.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:33.791+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:34.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:33.791+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:34.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:34.737+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:34.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:34.737+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:34.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:34.737+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:34.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:34.738+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:34.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:34.532+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:34.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:34.532+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:34.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:34.532+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:34.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:34.533+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:34.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:34.767+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:34.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:34.767+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:34.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:34.767+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:34.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:34.767+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:35.813 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:35.709+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:35.813 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:35.709+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:35.813 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:35.709+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:35.813 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:35.710+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:35.814 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:35.561+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:35.814 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:35.561+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:35.814 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:35.561+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:35.814 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:35.562+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:36.164 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:35.813+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:36.165 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:35.813+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:36.165 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:35.813+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:36.165 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:35.814+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:36.785 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:36.738+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:36.785 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:36.738+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:36.785 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:36.738+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:36.785 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:36.739+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:36.785 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:36.534+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:36.786 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:36.534+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:36.786 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:36.534+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:36.786 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:36.534+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:37.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:36.785+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:37.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:36.785+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:37.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:36.785+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:37.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:36.786+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:37.815 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:37.531+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:37.816 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:37.531+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:37.816 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:37.531+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:37.816 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:37.531+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:37.816 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:37.767+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:37.816 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:37.767+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:37.816 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:37.767+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:37.816 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:37.768+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:38.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:37.816+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:38.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:37.816+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:38.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:37.816+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:38.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:37.816+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:38.800 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:25:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:25:38.632+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 33575 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:25:38.801 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:38.801+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:38.801 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:38.801+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:38.801 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:38.801+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:38.801 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:38.801+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:38.801 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:38.551+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:38.802 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:38.551+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:38.802 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:38.551+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:38.802 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:38.551+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:39.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:38.853+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:39.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:38.853+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:39.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:38.853+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:39.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:38.853+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:39.767 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:39.502+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:39.767 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:39.502+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:39.767 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:39.502+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:39.767 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:39.502+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:40.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:39.767+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:40.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:39.767+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:40.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:39.767+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:40.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:39.768+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:40.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:39.807+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:40.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:39.807+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:40.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:39.807+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:40.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:39.807+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:40.794 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:40.524+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:40.795 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:40.524+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:40.795 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:40.524+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:40.795 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:40.524+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:40.795 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:40.767+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:40.795 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:40.767+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:40.796 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:40.767+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:40.796 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:40.768+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:41.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:40.795+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:41.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:40.795+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:41.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:40.795+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:41.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:40.795+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:41.837 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:41.486+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:41.838 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:41.486+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:41.838 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:41.486+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:41.838 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:41.486+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:41.838 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:41.728+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:41.839 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:41.728+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:41.839 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:41.728+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:41.839 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:41.729+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:42.134 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:41.838+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:42.134 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:41.838+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:42.134 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:41.838+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:42.134 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:41.838+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:42.834 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:42.492+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:42.834 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:42.492+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:42.834 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:42.492+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:42.835 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:42.492+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:42.835 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:42.718+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:42.835 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:42.718+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:42.835 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:42.718+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:42.835 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:42.718+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:43.188 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:42.834+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:43.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:42.834+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:43.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:42.834+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:43.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:42.835+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:43.762 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:25:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:25:43.638+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 33666 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:25:43.763 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:43.741+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:43.763 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:43.741+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:43.763 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:43.741+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:43.763 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:43.741+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:43.764 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:43.501+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:43.764 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:43.501+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:43.764 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:43.501+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:43.764 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:43.501+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:44.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:43.823+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:44.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:43.823+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:44.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:43.823+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:44.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:43.823+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:44.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:44.546+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:44.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:44.546+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:44.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:44.546+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:44.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:44.547+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:44.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:44.792+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:44.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:44.792+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:44.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:44.792+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:44.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:44.792+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:44.941 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:44.705+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:44.941 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:44.705+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:44.941 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:44.705+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:44.941 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:44.705+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:45.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:45.525+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:45.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:45.525+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:45.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:45.525+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:45.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:45.525+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:45.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:45.718+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:45.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:45.718+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:45.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:45.718+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:45.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:45.718+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:45.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:45.746+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:45.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:45.746+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:45.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:45.746+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:45.941 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:45.746+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:46.763 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:46.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:46.763 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:46.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:46.763 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:46.500+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:46.763 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:46.501+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:47.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:46.763+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:47.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:46.763+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:47.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:46.763+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:47.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:46.764+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:47.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:46.782+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:47.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:46.782+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:47.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:46.782+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:47.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:46.782+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:47.809 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:47.512+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:47.809 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:47.512+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:47.810 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:47.512+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:47.810 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:47.513+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:48.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:47.810+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:48.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:47.810+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:48.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:47.810+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:48.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:47.810+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:48.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:47.821+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:48.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:47.821+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:48.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:47.821+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:48.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:47.821+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:48.767 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:25:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:25:48.644+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 33755 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:25:48.768 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:48.491+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:48.768 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:48.491+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:48.768 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:48.491+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:48.768 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:48.491+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:49.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:48.775+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:49.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:48.775+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:49.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:48.775+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:49.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:48.775+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:49.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:48.864+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:49.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:48.864+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:49.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:48.864+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:49.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:48.864+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:49.813 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:49.461+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:49.813 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:49.461+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:49.813 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:49.461+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:49.813 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:49.461+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:50.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:49.813+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:50.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:49.813+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:50.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:49.813+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:50.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:49.814+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:50.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:49.895+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:50.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:49.895+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:50.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:49.895+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:50.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:49.895+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:50.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:50.417+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:50.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:50.417+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:50.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:50.417+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:50.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:50.417+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:51.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:50.809+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:51.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:50.809+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:51.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:50.809+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:51.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:50.809+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:51.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:50.915+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:51.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:50.915+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:51.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:50.915+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:51.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:50.916+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:51.522 ERROR:paramiko.transport:Socket exception: No route to host (113) 2024-04-06T03:25:51.527 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T03:25:51.528 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-04-06T03:25:51.529 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-04-06T03:25:51.544 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-06T03:25:51.562 INFO:tasks.cephadm:Teardown begin 2024-04-06T03:25:51.562 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-04-06T03:25:51.567 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-06T03:25:51.601 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:51.368+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:51.602 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:51.368+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:51.602 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:51.368+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:51.602 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:51.369+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:51.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:25:51.607 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:25:51.879 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:51.776+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:51.879 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:51.776+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:51.879 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:51.776+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:51.879 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:51.777+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:52.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:51.880+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:52.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:51.880+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:52.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:51.880+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:52.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:51.880+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:52.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:52.417+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:52.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:52.417+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:52.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:52.417+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:52.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:52.417+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:53.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:52.910+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:53.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:52.910+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:53.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:52.910+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:53.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:52.910+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:53.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:52.820+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:53.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:52.820+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:53.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:52.820+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:53.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:52.820+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:53.688 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:25:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:25:53.650+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 33846 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:25:53.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:53.435+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:53.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:53.435+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:53.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:53.435+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:53.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:53.435+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:54.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:53.807+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:54.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:53.807+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:54.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:53.807+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:54.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:53.807+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:54.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:53.917+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:54.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:53.917+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:54.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:53.917+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:54.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:53.917+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:54.686 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:25:54.688 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:54.419+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:54.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:54.419+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:54.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:54.419+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:54.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:54.419+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:55.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:54.906+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:55.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:54.906+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:55.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:54.906+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:55.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:54.906+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:55.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:54.828+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:55.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:54.828+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:55.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:54.828+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:55.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:54.829+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:55.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:55.383+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:55.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:55.383+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:55.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:55.383+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:55.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:55.383+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:56.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:55.866+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:56.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:55.866+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:56.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:55.866+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:56.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:55.866+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:56.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:55.860+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:56.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:55.860+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:56.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:55.860+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:56.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:55.861+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:56.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:56.368+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:56.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:56.368+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:56.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:56.368+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:56.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:56.369+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:57.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:56.833+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:57.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:56.833+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:57.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:56.833+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:57.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:56.833+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:57.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:56.878+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:57.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:56.878+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:57.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:56.878+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:57.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:56.878+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:57.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:57.346+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:57.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:57.346+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:57.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:57.346+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:57.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:57.346+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:58.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:57.817+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:58.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:57.817+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:58.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:57.817+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:58.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:57.817+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:58.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:57.829+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:58.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:57.829+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:58.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:57.829+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:58.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:57.829+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:58.656 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:58.373+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:58.656 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:58.373+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:58.657 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:58.373+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:58.657 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:58.373+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:58.939 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:25:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:25:58.656+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 33936 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:25:58.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:58.852+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:58.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:58.852+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:58.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:58.853+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:25:58.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:58.853+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:25:58.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:58.792+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:58.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:58.792+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:58.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:58.792+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:25:58.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:58.793+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:25:59.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:59.347+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:59.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:59.347+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:59.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:59.347+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:25:59.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:25:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:25:59.347+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:00.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:59.831+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:00.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:59.831+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:00.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:59.831+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:00.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:25:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:25:59.831+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:00.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:59.800+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:00.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:59.800+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:00.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:59.800+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:00.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:25:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:25:59.800+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:00.563 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:00.376+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:00.563 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:00.376+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:00.564 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:00.376+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:00.564 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:00.376+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:00.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:00.793+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:00.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:00.793+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:00.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:00.793+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:00.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:00.794+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:00.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:00.776+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:00.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:00.776+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:00.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:00.776+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:00.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:00.776+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:01.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:01.373+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:01.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:01.373+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:01.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:01.373+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:01.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:01.373+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:02.014 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:01.746+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:02.014 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:01.746+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:02.014 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:01.746+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:02.015 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:01.746+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:02.015 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:01.780+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:02.015 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:01.780+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:02.015 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:01.780+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:02.015 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:01.780+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:02.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:02.364+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:02.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:02.364+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:02.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:02.364+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:02.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:02.365+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:03.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:02.792+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:03.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:02.792+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:03.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:02.792+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:03.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:02.792+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:03.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:02.720+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:03.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:02.720+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:03.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:02.720+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:03.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:02.720+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:03.661 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:03.382+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:03.661 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:03.382+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:03.661 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:03.382+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:03.661 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:03.382+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:03.688 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:26:03.688 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:26:03.938 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:26:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:26:03.663+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 34027 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:26:03.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:03.786+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:03.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:03.786+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:03.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:03.786+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:03.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:03.787+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:03.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:03.723+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:03.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:03.723+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:03.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:03.723+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:03.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:03.723+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:04.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:04.404+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:04.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:04.404+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:04.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:04.404+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:04.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:04.404+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:05.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:04.746+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:05.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:04.746+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:05.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:04.746+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:05.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:04.747+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:05.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:04.753+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:05.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:04.753+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:05.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:04.753+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:05.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:04.754+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:05.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:05.429+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:05.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:05.429+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:05.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:05.429+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:05.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:05.429+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:06.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:05.717+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:06.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:05.717+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:06.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:05.717+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:06.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:05.718+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:06.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:05.711+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:06.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:05.711+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:06.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:05.711+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:06.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:05.711+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:06.682 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:06.380+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:06.682 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:06.380+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:06.682 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:06.380+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:06.683 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:06.381+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:06.750 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:26:06.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:06.683+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:06.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:06.683+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:06.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:06.683+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:06.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:06.683+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:06.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:06.686+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:06.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:06.687+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:06.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:06.687+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:06.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:06.687+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:07.649 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:07.366+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:07.649 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:07.366+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:07.649 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:07.366+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:07.649 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:07.366+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:07.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:07.650+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:07.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:07.650+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:07.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:07.650+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:07.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:07.650+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:07.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:07.668+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:07.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:07.668+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:07.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:07.668+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:07.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:07.668+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:08.669 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:08.356+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:08.669 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:08.356+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:08.669 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:08.356+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:08.669 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:08.356+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:08.939 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:26:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:26:08.669+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 34115 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:26:08.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:08.670+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:08.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:08.670+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:08.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:08.670+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:08.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:08.670+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:08.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:08.709+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:08.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:08.709+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:08.941 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:08.709+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:08.941 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:08.709+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:09.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:09.326+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:09.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:09.326+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:09.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:09.326+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:09.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:09.326+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:10.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:09.701+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:10.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:09.701+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:10.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:09.701+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:10.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:09.701+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:10.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:09.759+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:10.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:09.759+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:10.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:09.759+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:10.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:09.759+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:10.670 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:10.283+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:10.670 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:10.283+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:10.670 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:10.283+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:10.670 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:10.283+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:10.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:10.719+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:10.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:10.719+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:10.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:10.719+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:10.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:10.720+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:10.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:10.671+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:10.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:10.671+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:10.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:10.671+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:10.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:10.671+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:11.632 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:11.333+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:11.632 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:11.333+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:11.633 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:11.333+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:11.633 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:11.333+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:11.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:11.633+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:11.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:11.633+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:11.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:11.633+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:11.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:11.633+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:11.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:11.736+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:11.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:11.736+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:11.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:11.736+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:11.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:11.736+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:12.606 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:12.298+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:12.606 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:12.298+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:12.606 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:12.298+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:12.606 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:12.298+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:12.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:12.710+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:12.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:12.710+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:12.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:12.710+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:12.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:12.710+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:12.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:12.607+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:12.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:12.607+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:12.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:12.607+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:12.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:12.607+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:13.614 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:13.331+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:13.614 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:13.331+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:13.614 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:13.331+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:13.615 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:13.331+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:13.939 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:26:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:26:13.675+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 34205 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:26:13.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:13.744+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:13.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:13.744+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:13.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:13.744+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:13.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:13.744+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:13.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:13.615+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:13.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:13.615+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:13.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:13.615+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:13.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:13.615+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:14.581 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:14.326+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:14.581 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:14.326+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:14.581 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:14.326+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:14.582 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:14.326+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:14.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:14.756+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:14.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:14.756+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:14.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:14.756+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:14.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:14.757+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:14.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:14.581+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:14.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:14.581+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:14.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:14.581+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:14.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:14.582+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:15.689 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:15.583+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:15.689 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:15.583+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:15.689 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:15.583+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:15.689 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:15.583+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:15.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:15.365+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:15.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:15.365+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:15.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:15.365+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:15.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:15.365+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:16.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:15.759+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:16.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:15.759+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:16.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:15.759+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:16.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:15.759+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:16.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:16.368+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:16.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:16.368+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:16.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:16.368+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:16.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:16.368+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:16.690 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:16.611+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:16.690 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:16.611+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:16.690 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:16.611+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:16.690 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:16.611+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:17.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:16.761+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:17.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:16.761+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:17.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:16.761+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:17.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:16.761+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:17.689 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:17.586+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:17.689 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:17.586+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:17.689 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:17.586+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:17.689 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:17.586+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:17.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:17.356+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:17.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:17.356+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:17.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:17.356+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:17.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:17.356+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:18.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:17.769+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:18.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:17.769+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:18.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:17.769+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:18.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:17.769+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:18.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:18.540+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:18.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:18.540+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:18.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:18.540+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:18.681 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:18.540+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:18.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:18.346+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:18.681 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:18.346+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:18.682 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:18.346+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:18.682 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:18.346+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:18.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:26:18.752 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:26:18.939 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:26:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:26:18.680+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 34295 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:26:18.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:18.816+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:18.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:18.816+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:18.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:18.816+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:18.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:18.816+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:19.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:19.348+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:19.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:19.348+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:19.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:19.348+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:19.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:19.349+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:19.690 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:19.517+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:19.690 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:19.517+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:19.690 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:19.517+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:19.690 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:19.517+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:20.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:19.858+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:20.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:19.858+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:20.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:19.858+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:20.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:19.859+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:20.547 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:20.340+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:20.548 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:20.340+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:20.548 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:20.340+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:20.548 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:20.341+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:20.841 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:20.547+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:20.841 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:20.547+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:20.841 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:20.547+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:20.841 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:20.548+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:21.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:20.841+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:21.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:20.841+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:21.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:20.841+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:21.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:20.842+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:21.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:21.363+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:21.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:21.363+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:21.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:21.363+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:21.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:21.364+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:21.691 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:21.567+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:21.691 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:21.567+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:21.691 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:21.567+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:21.691 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:21.567+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:21.822 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:26:22.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:21.869+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:22.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:21.869+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:22.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:21.869+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:22.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:21.869+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:22.558 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:22.327+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:22.558 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:22.327+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:22.558 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:22.327+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:22.558 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:22.327+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:22.914 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:22.558+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:22.914 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:22.558+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:22.915 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:22.558+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:22.915 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:22.558+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:23.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:22.912+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:23.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:22.912+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:23.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:22.912+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:23.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:22.913+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:23.608 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:23.316+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:23.608 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:23.316+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:23.608 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:23.316+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:23.608 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:23.316+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:23.903 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:26:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:26:23.687+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 34385 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:26:23.904 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:23.608+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:23.904 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:23.608+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:23.904 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:23.608+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:23.904 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:23.608+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:24.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:23.904+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:24.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:23.904+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:24.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:23.904+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:24.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:23.904+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:24.607 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:24.298+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:24.608 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:24.298+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:24.608 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:24.298+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:24.608 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:24.298+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:24.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:26:24.824 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:26:24.878 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:24.607+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:24.878 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:24.607+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:24.878 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:24.607+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:24.878 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:24.607+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:24.898 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:26:24.898 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-06T03:26:24.899 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-06T03:26:24.899 DEBUG:teuthology.orchestra.run.smithi105:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-06T03:26:24.918 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:26:24.918 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:26:25.140 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:24.878+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:25.140 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:24.878+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:25.140 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:24.878+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:25.140 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:24.878+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:25.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:25.289+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:25.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:25.289+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:25.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:25.289+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:25.440 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:25.289+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:25.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:25.859+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:25.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:25.859+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:25.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:25.859+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:25.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:25.860+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:25.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:25.654+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:25.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:25.654+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:25.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:25.654+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:25.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:25.655+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:26.589 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:26.335+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:26.589 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:26.335+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:26.590 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:26.335+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:26.590 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:26.336+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:26.886 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:26.695+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:26.886 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:26.695+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:26.887 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:26.695+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:26.887 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:26.696+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:27.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:26.886+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:27.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:26.886+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:27.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:26.886+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:27.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:26.887+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:27.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:27.318+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:27.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:27.318+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:27.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:27.318+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:27.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:27.318+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:27.998 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:26:28.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:27.848+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:28.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:27.848+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:28.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:27.848+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:28.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:27.848+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:28.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:27.703+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:28.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:27.703+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:28.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:27.703+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:28.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:27.703+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:28.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:28.320+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:28.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:28.320+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:28.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:28.320+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:28.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:28.321+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:29.189 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:26:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:26:28.693+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 34475 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:26:29.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:28.889+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:29.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:28.889+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:29.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:28.889+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:29.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:28.890+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:29.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:28.742+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:29.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:28.742+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:29.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:28.742+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:29.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:28.742+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:29.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:29.294+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:29.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:29.294+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:29.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:29.294+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:29.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:29.294+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:30.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:29.911+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:30.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:29.911+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:30.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:29.911+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:30.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:29.912+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:30.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:29.781+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:30.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:29.781+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:30.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:29.781+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:30.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:29.782+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:30.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:30.319+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:30.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:30.319+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:30.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:30.319+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:30.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:30.319+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:31.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:30.901+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:31.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:30.901+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:31.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:30.901+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:31.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:30.901+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:31.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:30.814+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:31.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:30.814+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:31.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:30.814+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:31.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:30.815+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:31.631 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:31.368+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:31.631 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:31.368+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:31.632 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:31.368+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:31.632 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:31.368+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:31.892 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:31.803+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:31.892 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:31.803+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:31.892 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:31.803+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:31.893 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:31.804+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:32.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:31.892+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:32.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:31.892+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:32.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:31.892+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:32.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:31.893+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:32.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:32.343+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:32.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:32.343+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:32.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:32.343+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:32.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:32.343+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:33.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:32.918+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:33.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:32.918+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:33.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:32.918+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:33.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:32.919+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:33.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:32.770+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:33.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:32.770+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:33.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:32.770+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:33.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:32.770+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:33.664 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:33.315+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:33.664 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:33.315+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:33.664 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:33.315+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:33.664 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:33.315+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:33.939 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:26:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:26:33.699+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 34566 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:26:33.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:33.890+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:33.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:33.890+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:33.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:33.890+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:33.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:33.890+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:33.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:33.762+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:33.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:33.762+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:33.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:33.762+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:33.941 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:33.762+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:34.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:34.302+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:34.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:34.302+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:34.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:34.302+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:34.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:34.303+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:35.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:34.896+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:35.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:34.896+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:35.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:34.896+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:35.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:34.896+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:35.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:34.722+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:35.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:34.722+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:35.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:34.722+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:35.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:34.722+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:35.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:35.353+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:35.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:35.353+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:35.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:35.353+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:35.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:35.353+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:36.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:35.888+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:36.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:35.888+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:36.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:35.888+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:36.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:35.888+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:36.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:35.772+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:36.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:35.772+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:36.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:35.772+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:36.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:35.772+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:36.673 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:36.399+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:36.673 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:36.399+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:36.674 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:36.399+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:36.674 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:36.400+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:36.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:36.839+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:36.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:36.839+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:36.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:36.839+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:36.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:36.839+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:36.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:36.818+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:36.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:36.818+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:36.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:36.818+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:36.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:36.818+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:36.999 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:26:37.000 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:26:37.804 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:37.448+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:37.804 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:37.448+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:37.804 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:37.448+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:37.804 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:37.448+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:38.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:37.806+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:38.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:37.806+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:38.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:37.806+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:38.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:37.806+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:38.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:37.804+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:38.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:37.804+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:38.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:37.804+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:38.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:37.804+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:38.705 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:38.442+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:38.705 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:38.442+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:38.705 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:38.442+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:38.706 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:38.443+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:39.005 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:26:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:26:38.705+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 34657 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:26:39.005 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:38.826+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:39.005 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:38.826+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:39.005 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:38.826+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:39.006 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:38.827+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:39.006 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:38.855+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:39.006 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:38.855+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:39.006 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:38.855+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:39.006 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:38.855+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:39.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:39.431+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:39.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:39.431+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:39.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:39.431+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:39.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:39.432+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:40.062 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:26:40.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:39.866+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:40.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:39.866+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:40.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:39.866+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:40.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:39.866+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:40.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:39.851+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:40.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:39.851+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:40.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:39.851+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:40.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:39.851+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:40.499 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:40.383+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:40.500 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:40.383+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:40.500 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:40.383+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:40.500 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:40.383+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:41.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:40.882+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:41.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:40.882+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:41.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:40.882+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:41.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:40.882+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:41.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:40.868+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:41.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:40.868+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:41.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:40.868+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:41.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:40.868+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:41.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:41.391+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:41.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:41.391+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:41.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:41.391+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:41.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:41.391+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:42.014 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:41.884+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:42.015 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:41.884+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:42.015 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:41.884+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:42.015 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:41.884+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:42.015 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:41.928+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:42.015 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:41.928+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:42.015 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:41.928+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:42.016 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:41.928+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:42.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:42.409+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:42.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:42.409+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:42.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:42.409+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:42.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:42.409+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:43.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:42.900+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:43.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:42.900+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:43.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:42.900+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:43.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:42.900+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:43.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:42.900+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:43.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:42.900+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:43.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:42.900+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:43.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:42.900+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:43.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:43.438+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:43.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:43.438+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:43.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:43.438+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:43.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:43.438+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:44.189 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:26:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:26:43.711+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 34748 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:26:44.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:43.942+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:44.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:43.942+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:44.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:43.942+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:44.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:43.942+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:44.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:43.899+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:44.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:43.899+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:44.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:43.899+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:44.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:43.899+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:44.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:44.395+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:44.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:44.395+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:44.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:44.395+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:44.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:44.395+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:45.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:44.928+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:45.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:44.928+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:45.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:44.928+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:45.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:44.928+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:45.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:44.894+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:45.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:44.894+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:45.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:44.894+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:45.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:44.894+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:45.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:45.363+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:45.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:45.363+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:45.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:45.363+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:45.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:45.364+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:46.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:45.896+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:46.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:45.896+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:46.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:45.896+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:46.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:45.896+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:46.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:45.861+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:46.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:45.861+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:46.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:45.861+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:46.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:45.861+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:46.644 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:46.374+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:46.644 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:46.374+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:46.644 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:46.374+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:46.644 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:46.374+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:46.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:46.882+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:46.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:46.882+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:46.939 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:46.882+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:46.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:46.882+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:46.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:46.827+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:46.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:46.827+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:46.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:46.827+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:46.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:46.827+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:47.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:47.371+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:47.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:47.371+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:47.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:47.371+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:47.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:47.371+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:48.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:47.860+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:48.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:47.860+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:48.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:47.860+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:48.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:47.860+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:48.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:47.807+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:48.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:47.807+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:48.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:47.807+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:48.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:47.807+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:48.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:48.384+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:48.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:48.384+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:48.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:48.384+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:48.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:48.384+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:49.189 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:26:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:26:48.718+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 34837 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:26:49.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:48.858+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:49.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:48.858+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:49.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:48.858+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:49.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:48.859+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:49.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:48.845+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:49.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:48.845+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:49.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:48.845+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:49.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:48.845+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:49.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:49.415+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:49.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:49.415+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:49.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:49.415+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:49.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:49.415+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:49.948 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:49.864+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:49.948 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:49.864+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:49.948 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:49.864+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:49.948 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:49.864+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:49.949 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:49.856+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:49.949 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:49.856+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:49.949 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:49.856+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:49.949 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:49.856+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:50.500 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:50.404+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:50.500 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:50.404+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:50.501 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:50.404+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:50.501 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:50.404+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:51.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:50.826+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:51.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:50.826+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:51.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:50.826+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:51.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:50.826+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:51.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:50.867+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:51.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:50.867+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:51.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:50.867+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:51.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:50.867+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:51.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:51.454+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:51.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:51.454+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:51.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:51.454+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:51.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:51.454+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:52.063 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:26:52.064 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:26:52.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:51.884+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:52.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:51.884+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:52.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:51.884+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:52.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:51.884+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:52.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:51.796+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:52.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:51.796+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:52.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:51.796+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:52.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:51.796+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:52.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:52.406+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:52.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:52.406+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:52.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:52.406+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:52.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:52.406+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:53.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:52.774+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:53.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:52.774+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:53.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:52.774+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:53.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:52.775+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:53.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:52.920+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:53.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:52.920+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:53.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:52.920+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:53.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:52.921+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:53.664 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:53.382+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:53.665 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:53.382+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:53.665 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:53.382+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:53.665 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:53.382+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:53.939 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:26:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:26:53.723+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 34929 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:26:53.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:53.812+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:53.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:53.812+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:53.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:53.812+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:53.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:53.813+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:54.417 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:53.943+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:54.417 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:53.943+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:54.417 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:53.943+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:54.417 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:53.943+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:54.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:54.418+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:54.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:54.418+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:54.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:54.418+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:54.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:54.418+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:55.134 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:26:55.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:54.969+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:55.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:54.969+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:55.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:54.969+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:55.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:54.969+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:55.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:54.831+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:55.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:54.831+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:55.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:54.831+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:55.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:54.831+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:55.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:55.419+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:55.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:55.419+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:55.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:55.419+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:55.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:55.419+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:56.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:55.996+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:56.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:55.996+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:56.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:55.996+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:56.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:55.996+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:56.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:55.870+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:56.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:55.870+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:56.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:55.870+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:56.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:55.870+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:56.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:56.399+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:56.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:56.399+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:56.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:56.399+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:56.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:56.399+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:57.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:57.037+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:57.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:57.037+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:57.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:57.037+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:57.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:57.038+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:57.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:56.856+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:57.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:56.856+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:57.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:56.856+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:57.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:56.856+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:57.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:57.410+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:57.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:57.410+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:57.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:57.410+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:57.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:57.410+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:58.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:26:58.136 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:26:58.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:58.085+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:58.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:58.085+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:58.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:58.085+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:58.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:58.085+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:58.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:57.851+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:58.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:57.851+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:58.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:57.851+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:58.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:57.851+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:58.206 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:26:58.206 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-06T03:26:58.207 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-06T03:26:58.209 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641871/remote/smithi105/crash 2024-04-06T03:26:58.210 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/crash -- . 2024-04-06T03:26:58.246 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/lib/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641871/remote/smithi184/crash 2024-04-06T03:26:58.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:26:58.247 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:26:58.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:58.421+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:58.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:58.421+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:58.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:58.421+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:58.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:58.421+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:59.049 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:26:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:26:58.730+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 35019 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:26:59.050 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:58.850+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:59.050 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:58.850+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:59.050 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:58.850+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:26:59.050 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:58.850+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:26:59.413 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:59.050+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:59.414 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:59.050+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:59.414 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:59.050+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:26:59.414 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:26:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:26:59.050+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:26:59.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:59.414+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:59.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:59.414+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:59.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:59.414+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:26:59.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:26:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:26:59.414+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:00.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:00.084+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:00.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:00.084+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:00.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:00.084+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:00.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:00.084+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:00.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:59.839+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:00.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:59.839+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:00.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:59.839+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:00.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:26:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:26:59.839+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:00.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:00.364+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:00.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:00.364+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:00.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:00.364+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:00.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:00.365+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:01.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:01.037+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:01.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:01.037+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:01.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:01.037+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:01.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:01.037+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:01.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:00.885+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:01.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:00.885+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:01.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:00.885+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:01.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:00.885+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:01.310 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:27:01.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:01.387+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:01.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:01.387+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:01.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:01.387+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:01.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:01.387+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:02.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:02.056+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:02.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:02.056+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:02.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:02.056+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:02.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:02.057+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:02.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:01.839+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:02.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:01.839+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:02.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:01.839+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:02.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:01.839+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:02.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:02.350+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:02.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:02.350+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:02.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:02.350+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:02.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:02.350+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:03.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:03.096+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:03.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:03.096+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:03.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:03.096+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:03.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:03.096+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:03.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:02.874+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:03.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:02.874+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:03.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:02.874+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:03.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:02.874+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:03.665 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:03.332+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:03.665 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:03.332+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:03.665 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:03.332+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:03.665 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:03.332+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:03.939 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:27:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:27:03.735+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 35110 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:27:03.939 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:03.842+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:03.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:03.842+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:03.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:03.842+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:03.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:03.842+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:04.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:04.312+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:04.440 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:04.312+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:04.440 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:04.312+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:04.440 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:04.312+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:04.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:04.123+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:04.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:04.123+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:04.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:04.123+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:04.441 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:04.124+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:05.078 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:04.812+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:05.078 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:04.812+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:05.079 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:04.812+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:05.079 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:04.812+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:05.340 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:05.105+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:05.340 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:05.105+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:05.341 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:05.105+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:05.341 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:05.105+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:05.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:05.341+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:05.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:05.341+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:05.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:05.341+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:05.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:05.341+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:06.109 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:05.798+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:06.110 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:05.798+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:06.110 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:05.798+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:06.110 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:05.798+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:06.382 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:06.109+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:06.382 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:06.109+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:06.382 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:06.109+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:06.382 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:06.109+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:06.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:06.381+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:06.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:06.381+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:06.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:06.381+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:06.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:06.381+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:07.139 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:06.834+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:07.140 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:06.834+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:07.140 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:06.834+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:07.140 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:06.834+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:07.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:07.352+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:07.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:07.352+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:07.440 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:07.352+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:07.440 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:07.352+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:07.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:07.139+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:07.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:07.139+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:07.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:07.139+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:07.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:07.139+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:08.135 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:07.793+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:08.135 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:07.793+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:08.135 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:07.793+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:08.135 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:07.793+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:08.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:08.311+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:08.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:08.311+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:08.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:08.311+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:08.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:08.312+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:08.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:08.135+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:08.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:08.135+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:08.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:08.135+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:08.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:08.135+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:09.129 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:27:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:27:08.741+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 35200 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:27:09.130 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:08.778+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:09.130 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:08.778+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:09.130 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:08.778+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:09.130 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:08.779+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:09.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:09.344+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:09.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:09.344+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:09.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:09.344+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:09.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:09.345+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:09.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:09.129+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:09.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:09.129+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:09.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:09.129+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:09.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:09.130+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:10.089 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:09.815+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:10.089 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:09.815+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:10.089 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:09.815+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:10.090 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:09.815+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:10.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:27:10.312 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:27:10.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:10.331+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:10.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:10.331+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:10.440 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:10.331+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:10.440 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:10.331+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:10.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:10.089+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:10.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:10.089+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:10.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:10.089+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:10.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:10.089+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:11.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:11.056+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:11.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:11.056+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:11.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:11.056+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:11.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:11.056+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:11.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:10.851+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:11.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:10.851+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:11.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:10.851+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:11.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:10.852+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:11.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:11.369+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:11.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:11.369+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:11.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:11.369+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:11.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:11.370+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:12.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:12.102+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:12.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:12.102+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:12.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:12.102+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:12.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:12.103+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:12.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:11.857+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:12.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:11.857+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:12.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:11.857+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:12.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:11.857+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:12.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:12.400+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:12.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:12.400+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:12.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:12.400+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:12.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:12.400+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:13.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:13.067+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:13.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:13.067+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:13.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:13.067+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:13.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:13.068+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:13.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:12.848+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:13.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:12.848+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:13.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:12.848+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:13.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:12.848+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:13.374 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:27:13.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:13.434+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:13.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:13.434+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:13.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:13.434+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:13.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:13.435+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:14.056 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:27:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:27:13.748+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 35292 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:27:14.056 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:13.856+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:14.056 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:13.856+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:14.056 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:13.856+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:14.056 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:13.857+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:14.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:14.056+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:14.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:14.056+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:14.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:14.056+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:14.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:14.056+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:14.847 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:14.450+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:14.847 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:14.450+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:14.847 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:14.450+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:14.847 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:14.451+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:15.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:15.024+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:15.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:15.024+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:15.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:15.024+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:15.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:15.025+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:15.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:14.847+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:15.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:14.847+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:15.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:14.847+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:15.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:14.847+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:15.816 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:15.452+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:15.816 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:15.452+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:15.816 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:15.452+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:15.816 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:15.452+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:16.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:16.048+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:16.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:16.048+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:16.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:16.048+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:16.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:16.049+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:16.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:15.816+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:16.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:15.816+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:16.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:15.816+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:16.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:15.816+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:16.846 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:16.460+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:16.846 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:16.460+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:16.846 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:16.460+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:16.846 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:16.460+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:17.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:17.030+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:17.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:17.030+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:17.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:17.030+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:17.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:17.030+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:17.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:16.846+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:17.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:16.846+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:17.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:16.846+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:17.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:16.846+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:17.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:17.416+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:17.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:17.416+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:17.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:17.416+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:17.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:17.417+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:18.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:18.007+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:18.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:18.007+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:18.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:18.007+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:18.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:18.007+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:18.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:17.844+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:18.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:17.844+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:18.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:17.844+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:18.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:17.844+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:18.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:18.427+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:18.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:18.427+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:18.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:18.427+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:18.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:18.427+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:19.017 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:27:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:27:18.754+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 35382 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:27:19.017 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:18.863+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:19.018 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:18.863+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:19.018 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:18.863+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:19.018 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:18.864+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:19.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:19.017+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:19.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:19.017+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:19.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:19.017+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:19.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:19.017+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:19.879 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:19.459+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:19.880 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:19.459+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:19.880 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:19.459+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:19.880 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:19.459+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:20.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:19.879+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:20.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:19.880+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:20.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:19.880+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:20.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:19.880+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:20.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:19.971+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:20.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:19.971+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:20.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:19.971+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:20.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:19.971+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:20.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:20.472+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:20.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:20.472+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:20.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:20.472+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:20.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:20.472+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:21.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:20.838+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:21.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:20.838+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:21.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:20.838+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:21.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:20.838+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:21.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:20.998+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:21.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:20.998+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:21.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:20.998+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:21.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:20.998+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:21.841 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:21.484+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:21.842 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:21.484+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:21.842 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:21.484+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:21.842 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:21.484+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:22.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:21.841+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:22.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:21.841+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:22.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:21.841+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:22.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:21.841+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:22.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:21.971+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:22.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:21.971+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:22.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:21.971+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:22.191 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:21.971+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:22.812 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:22.473+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:22.812 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:22.473+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:22.812 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:22.473+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:22.813 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:22.473+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:23.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:22.812+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:23.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:22.812+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:23.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:22.812+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:23.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:22.812+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:23.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:22.970+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:23.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:22.970+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:23.191 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:22.970+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:23.191 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:22.970+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:23.768 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:27:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:27:23.760+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 35473 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:27:23.769 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:23.515+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:23.769 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:23.515+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:23.769 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:23.515+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:23.769 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:23.515+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:24.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:23.841+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:24.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:23.841+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:24.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:23.841+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:24.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:23.841+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:24.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:24.001+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:24.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:24.001+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:24.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:24.001+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:24.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:24.001+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:24.824 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:24.520+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:24.824 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:24.520+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:24.824 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:24.520+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:24.824 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:24.521+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:25.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:25.002+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:25.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:25.002+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:25.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:25.002+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:25.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:25.002+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:25.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:24.824+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:25.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:24.824+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:25.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:24.824+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:25.192 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:24.824+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:25.376 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:27:25.376 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:27:25.807 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:25.523+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:25.807 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:25.523+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:25.807 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:25.523+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:25.808 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:25.523+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:26.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:26.018+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:26.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:26.018+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:26.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:26.018+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:26.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:26.018+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:26.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:25.807+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:26.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:25.807+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:26.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:25.807+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:26.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:25.807+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:26.850 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:26.525+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:26.850 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:26.525+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:26.850 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:26.525+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:26.851 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:26.525+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:27.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:26.973+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:27.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:26.973+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:27.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:26.973+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:27.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:26.973+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:27.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:26.850+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:27.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:26.850+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:27.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:26.850+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:27.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:26.850+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:27.838 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:27.543+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:27.839 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:27.543+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:27.839 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:27.543+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:27.839 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:27.543+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:28.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:27.838+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:28.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:27.838+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:28.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:27.838+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:28.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:27.838+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:28.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:27.964+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:28.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:27.964+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:28.191 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:27.964+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:28.191 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:27.965+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:28.450 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:27:28.776 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:27:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:27:28.766+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 35563 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:27:28.776 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:28.525+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:28.777 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:28.525+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:28.777 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:28.525+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:28.777 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:28.525+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:29.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:28.868+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:29.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:28.868+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:29.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:28.868+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:29.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:28.868+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:29.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:28.963+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:29.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:28.963+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:29.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:28.963+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:29.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:28.963+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:29.838 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:29.535+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:29.838 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:29.535+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:29.838 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:29.535+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:29.838 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:29.535+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:30.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:29.837+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:30.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:29.837+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:30.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:29.837+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:30.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:29.838+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:30.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:29.977+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:30.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:29.977+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:30.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:29.977+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:30.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:29.977+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:30.879 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:30.526+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:30.879 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:30.526+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:30.879 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:30.526+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:30.879 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:30.526+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:31.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:30.879+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:31.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:30.879+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:31.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:30.879+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:31.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:30.879+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:31.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:31.018+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:31.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:31.018+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:31.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:31.018+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:31.191 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:31.018+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:31.452 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:27:31.452 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:27:31.518 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:27:31.518 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-06T03:27:31.519 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-06T03:27:31.519 DEBUG:teuthology.orchestra.run.smithi105:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-04-06T03:27:31.562 INFO:tasks.cephadm:Compressing logs... 2024-04-06T03:27:31.563 DEBUG:teuthology.orchestra.run.smithi105:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-06T03:27:31.605 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:27:31.606 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:27:31.632 INFO:teuthology.orchestra.run.smithi105.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-06T03:27:31.633 INFO:teuthology.orchestra.run.smithi105.stderr:: No such file or directory 2024-04-06T03:27:31.633 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-mds.cephfs.smithi105.oxxcbq.log 2024-04-06T03:27:31.634 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph.audit.log 2024-04-06T03:27:31.634 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-mon.smithi105.log 2024-04-06T03:27:31.634 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-mds.cephfs.smithi105.oxxcbq.log: /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-client.ceph-exporter.smithi105.log 2024-04-06T03:27:31.634 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose --/var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-mon.smithi105.log: /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-osd.1.log 2024-04-06T03:27:31.635 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-client.ceph-exporter.smithi105.log: gzip -5 --verbose -- /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-mds.cephfs.smithi105.mrivth.log 2024-04-06T03:27:31.636 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-mgr.smithi105.vqqcpx.log 2024-04-06T03:27:31.637 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-mds.cephfs.smithi105.mrivth.log: gzip -5 --verbose -- /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-osd.0.log 2024-04-06T03:27:31.637 INFO:teuthology.orchestra.run.smithi105.stderr: 90.7% -- replaced with /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph.audit.log.gz 2024-04-06T03:27:31.637 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-mgr.smithi105.vqqcpx.log: gzip -5 --verbose -- /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph.cephadm.log 2024-04-06T03:27:31.637 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-osd.0.log: 94.8% -- replaced with /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-client.ceph-exporter.smithi105.log.gz 2024-04-06T03:27:31.644 INFO:teuthology.orchestra.run.smithi105.stderr: 90.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-06T03:27:31.649 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph.log 2024-04-06T03:27:31.649 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-osd.2.log 2024-04-06T03:27:31.650 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph.log: 83.6% -- replaced with /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph.cephadm.log.gz 2024-04-06T03:27:31.650 INFO:teuthology.orchestra.run.smithi105.stderr: 87.1% -- replaced with /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph.log.gz 2024-04-06T03:27:31.651 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-volume.log 2024-04-06T03:27:31.659 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-osd.2.log: /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-volume.log: 90.8% -- replaced with /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-volume.log.gz 2024-04-06T03:27:31.671 INFO:teuthology.orchestra.run.smithi105.stderr: 95.0% -- replaced with /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-mds.cephfs.smithi105.mrivth.log.gz 2024-04-06T03:27:31.858 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:31.533+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:31.859 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:31.533+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:31.859 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:31.533+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:31.859 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:31.533+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:31.859 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:31.857+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:32.135 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:32.049+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:32.136 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:32.049+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:32.136 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:32.049+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:32.136 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:32.050+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:32.136 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:31.857+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:32.136 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:31.858+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:32.137 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:31.858+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:32.551 INFO:teuthology.orchestra.run.smithi105.stderr:gzip: /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-mgr.smithi105.vqqcpx.log: file size changed while zipping 2024-04-06T03:27:32.551 INFO:teuthology.orchestra.run.smithi105.stderr: 89.8% -- replaced with /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-mgr.smithi105.vqqcpx.log.gz 2024-04-06T03:27:32.886 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:32.526+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:32.886 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:32.526+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:32.886 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:32.526+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:32.886 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:32.526+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:33.144 INFO:teuthology.orchestra.run.smithi105.stderr:gzip: /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-mon.smithi105.log: file size changed while zipping 2024-04-06T03:27:33.144 INFO:teuthology.orchestra.run.smithi105.stderr: 92.6% -- replaced with /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-mon.smithi105.log.gz 2024-04-06T03:27:33.182 INFO:teuthology.orchestra.run.smithi105.stderr:gzip: /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-osd.2.log: file size changed while zipping 2024-04-06T03:27:33.182 INFO:teuthology.orchestra.run.smithi105.stderr: 93.4% -- replaced with /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-osd.2.log.gz 2024-04-06T03:27:33.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:33.031+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:33.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:33.031+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:33.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:33.031+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:33.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:33.031+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:33.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:32.885+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:33.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:32.885+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:33.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:32.885+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:33.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:32.886+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:33.825 INFO:teuthology.orchestra.run.smithi105.stderr:gzip: /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-osd.1.log: file size changed while zipping 2024-04-06T03:27:33.825 INFO:teuthology.orchestra.run.smithi105.stderr: 92.7% -- replaced with /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-osd.1.log.gz 2024-04-06T03:27:33.864 INFO:teuthology.orchestra.run.smithi105.stderr:gzip: /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-osd.0.log: file size changed while zipping 2024-04-06T03:27:33.864 INFO:teuthology.orchestra.run.smithi105.stderr: 93.0% -- replaced with /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-osd.0.log.gz 2024-04-06T03:27:33.898 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:27:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:27:33.770+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 35655 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:27:33.898 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:33.555+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:33.899 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:33.555+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:33.899 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:33.555+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:33.899 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:33.556+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:34.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:34.026+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:34.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:34.026+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:34.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:34.026+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:34.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:34.026+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:34.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:33.933+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:34.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:33.933+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:34.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:33.933+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:34.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:33.933+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:34.686 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:27:34.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:34.535+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:34.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:34.535+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:34.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:34.535+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:34.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:34.535+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:35.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:35.049+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:35.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:35.049+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:35.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:35.049+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:35.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:35.050+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:35.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:34.963+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:35.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:34.963+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:35.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:34.963+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:35.441 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:34.963+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:35.796 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:35.488+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:35.806 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:35.488+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:35.807 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:35.488+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:35.807 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:35.488+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:36.046 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:35.994+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:36.046 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:35.994+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:36.047 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:35.994+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:36.047 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:35.994+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:36.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:36.046+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:36.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:36.046+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:36.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:36.046+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:36.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:36.047+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:36.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:36.518+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:36.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:36.518+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:36.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:36.518+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:36.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:36.519+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:37.202 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:37.074+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:37.202 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:37.074+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:37.202 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:37.074+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:37.202 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:37.074+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:37.203 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:36.945+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:37.203 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:36.945+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:37.203 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:36.945+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:37.203 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:36.945+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:37.936 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:37.527+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:37.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:37.527+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:37.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:37.527+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:37.937 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:37.527+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:38.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:38.032+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:38.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:38.032+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:38.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:38.032+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:38.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:38.032+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:38.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:37.936+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:38.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:37.936+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:38.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:37.936+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:38.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:37 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:37.936+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:38.786 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:27:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:27:38.775+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 35746 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:27:38.787 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:38.528+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:38.787 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:38.528+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:38.787 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:38.528+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:38.787 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:38.528+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:39.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:38.987+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:39.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:38.987+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:39.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:38.987+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:39.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:38.987+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:39.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:38.973+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:39.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:38.973+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:39.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:38.973+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:39.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:38 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:38.973+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:39.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:39.519+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:39.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:39.519+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:39.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:39.519+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:39.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:39.519+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:40.239 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:39.959+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:40.239 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:39.959+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:40.240 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:39.959+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:40.240 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:39.960+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:40.240 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:39.996+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:40.240 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:39.996+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:40.240 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:39.996+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:40.240 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:39 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:39.996+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:40.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:40.554+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:40.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:40.554+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:40.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:40.554+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:40.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:40.554+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:41.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:40.991+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:41.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:40.991+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:41.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:40.991+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:41.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:40.991+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:41.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:40.969+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:41.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:40.969+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:41.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:40.969+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:41.441 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:40 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:40.969+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:41.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:41.521+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:41.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:41.521+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:41.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:41.521+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:41.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:41.521+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:42.241 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:41.961+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:42.241 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:41.961+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:42.242 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:41.961+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:42.242 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:41 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:41.961+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:42.242 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:42.005+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:42.242 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:42.005+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:42.242 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:42.005+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:42.242 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:42.005+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:42.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:42.567+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:42.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:42.567+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:42.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:42.567+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:42.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:42.568+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:43.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:42.982+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:43.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:42.982+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:43.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:42.982+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:43.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:42 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:42.983+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:43.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:43.018+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:43.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:43.018+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:43.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:43.018+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:43.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:43.019+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:43.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:27:43.688 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:27:43.904 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:27:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:27:43.779+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 35838 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:27:43.904 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:43.567+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:43.904 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:43.567+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:43.905 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:43.567+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:43.905 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:43.568+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:44.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:43.993+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:44.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:43.993+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:44.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:43.993+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:44.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:43 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:43.994+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:44.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:44.008+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:44.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:44.008+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:44.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:44.008+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:44.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:44.008+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:44.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:44.572+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:44.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:44.572+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:44.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:44.572+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:44.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:44.573+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:45.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:44.981+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:45.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:44.981+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:45.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:44.981+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:45.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:44 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:44.981+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:45.441 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:45.005+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:45.441 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:45.005+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:45.441 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:45.005+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:45.441 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:45.005+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:45.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:45.596+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:45.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:45.596+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:45.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:45.596+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:45.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:45.596+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:46.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:45.951+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:46.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:45.951+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:46.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:45.951+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:46.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:45.952+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:46.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:45.984+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:46.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:45.984+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:46.441 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:45.984+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:46.441 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:45 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:45.984+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:46.750 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:27:46.914 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:46.598+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:46.914 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:46.598+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:46.914 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:46.598+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:46.914 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:46.599+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:47.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:46.914+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:47.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:46.914+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:47.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:46.914+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:47.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:46.914+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:47.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:46.995+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:47.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:46.995+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:47.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:46.995+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:47.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:46 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:46.995+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:47.783 INFO:teuthology.orchestra.run.smithi105.stderr: 92.2% -- replaced with /var/log/ceph/130df578-f3bf-11ee-b649-cb9ed24678a4/ceph-mds.cephfs.smithi105.oxxcbq.log.gz 2024-04-06T03:27:47.785 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-06T03:27:47.785 INFO:teuthology.orchestra.run.smithi105.stderr:real 0m16.166s 2024-04-06T03:27:47.785 INFO:teuthology.orchestra.run.smithi105.stderr:user 0m23.902s 2024-04-06T03:27:47.785 INFO:teuthology.orchestra.run.smithi105.stderr:sys 0m0.665s 2024-04-06T03:27:47.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:47.593+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:47.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:47.593+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:47.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:47.593+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:47.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:47.593+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:48.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:47.950+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:48.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:47.950+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:48.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:47.950+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:48.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:47.950+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:48.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:47.948+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:48.441 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:47.948+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:48.441 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:47.948+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:48.441 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:47 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:47.948+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:48.909 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:27:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:27:48.786+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 35927 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:27:48.909 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:48.616+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:48.910 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:48.616+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:48.910 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:48.616+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:48.910 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:48.616+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:49.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:48.969+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:49.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:48.969+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:49.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:48.969+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:49.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:48.969+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:49.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:48.985+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:49.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:48.985+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:49.191 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:48.985+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:49.191 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:48 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:48.985+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:49.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:49.655+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:49.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:49.655+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:49.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:49.655+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:49.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:49 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:49.656+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:50.439 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:50.009+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:50.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:50.009+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:50.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:50.009+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:50.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:50.009+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:50.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:50.029+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:50.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:50.029+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:50.441 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:50.029+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:50.441 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:50.029+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:50.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:50.673+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:50.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:50.673+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:50.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:50.673+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:50.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:50.673+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:51.439 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:50.984+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:51.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:50.984+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:51.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:50.984+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:51.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:50 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:50.984+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:51.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:51.004+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:51.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:51.004+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:51.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:51.004+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:51.441 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:51.004+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:51.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:51.674+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:51.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:51.674+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:51.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:51.674+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:51.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:51.674+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:52.319 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:51.995+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:52.319 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:51.995+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:52.319 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:51.995+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:52.320 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:51.995+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:52.320 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:51.980+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:52.320 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:51.980+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:52.320 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:51.980+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:52.320 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:51 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:51.981+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:52.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:52.680+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:52.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:52.680+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:52.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:52.680+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:52.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:52.680+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:53.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:52.980+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:53.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:52.980+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:53.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:52.980+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:53.441 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:52 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:52.980+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:53.441 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:53.013+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:53.441 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:53.013+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:53.441 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:53.013+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:53.442 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:53.013+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:53.914 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:27:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:27:53.792+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 36015 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:27:53.915 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:53.636+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:53.915 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:53.636+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:53.915 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:53.636+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:53.915 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:53.636+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:54.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:53.970+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:54.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:53.970+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:54.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:53.970+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:54.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:53.970+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:54.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:53.944+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:54.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:53.944+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:54.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:53.944+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:54.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:53 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:53.944+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:54.921 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:54.671+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:54.921 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:54.671+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:54.921 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:54.671+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:54.921 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:54.671+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:55.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:54.920+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:55.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:54.920+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:55.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:54.920+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:55.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:54.920+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:55.191 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:54.926+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:55.191 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:54.926+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:55.191 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:54.926+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:55.191 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:54 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:54.927+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:56.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:55.895+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:56.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:55.895+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:56.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:55.895+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:56.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:55.895+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:56.190 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:55.705+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:56.191 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:55.705+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:56.191 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:55.705+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:56.191 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:55.705+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:56.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:55.922+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:56.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:55.922+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:56.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:55.922+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:56.192 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:55 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:55.922+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:56.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:56.687+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:56.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:56.687+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:56.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:56.688+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:56.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:56.688+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:56.940 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:56.936+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:56.941 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:56.936+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:56.941 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:56.936+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:56.941 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:56.936+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:57.362 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:56.946+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:57.363 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:56.946+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:57.363 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:56.946+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:57.363 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:56 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:56.946+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:58.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:57.914+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:58.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:57.914+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:58.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:57.914+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:58.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:57.915+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:58.191 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:57.713+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:58.191 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:57.713+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:58.191 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:57.713+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:58.191 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:57.714+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:58.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:57.950+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:58.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:57.950+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:58.192 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:57.950+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:58.192 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:57 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:57.950+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:58.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:27:58.752 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:27:58.972 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:27:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:27:58.799+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 36106 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:27:58.973 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:58.718+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:58.973 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:58.718+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:58.973 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:58.718+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:58.973 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:58.718+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:58.974 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:58.947+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:58.974 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:58.948+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:58.974 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:58.948+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:27:58.974 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:58.948+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:27:59.439 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:58.972+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:59.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:58.972+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:59.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:58.972+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:59.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:58 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:58.972+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:59.979 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:59.718+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:59.980 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:59.718+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:59.980 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:59.718+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:27:59.980 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:27:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:27:59.718+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:27:59.980 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:59.966+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:59.981 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:59.966+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:59.981 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:59.966+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:27:59.981 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:27:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:27:59.966+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:00.241 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:59.979+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:00.242 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:59.979+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:00.242 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:59.979+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:00.242 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:27:59 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:27:59.980+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:00.998 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:00.727+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:00.998 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:00.727+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:00.998 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:00.727+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:00.998 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:00.728+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:00.999 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:00.932+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:00.999 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:00.932+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:00.999 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:00.932+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:00.999 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:00.932+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:01.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:00.997+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:01.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:00.997+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:01.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:00.997+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:01.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:00 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:00.997+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:01.826 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:28:02.016 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:01.698+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:02.016 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:01.698+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:02.016 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:01.698+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:02.017 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:01.698+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:02.017 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:01.918+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:02.017 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:01.918+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:02.017 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:01.918+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:02.017 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:01 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:01.919+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:02.405 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:02.036+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:02.405 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:02.036+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:02.405 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:02.036+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:02.405 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:02.036+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:02.961 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:02.694+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:02.961 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:02.694+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:02.962 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:02.694+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:02.962 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:02.694+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:03.439 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:02.961+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:03.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:02.961+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:03.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:02.961+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:03.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:02 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:02.961+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:03.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:03.044+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:03.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:03.044+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:03.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:03.044+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:03.441 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:03.045+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:03.927 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:28:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:28:03.805+0000 7fd048e09700 -1 mon.smithi105@0(probing) e2 get_health_metrics reporting 36198 slow ops, oldest is monmgrreport(gid 14221, 0 checks, 1 progress events) 2024-04-06T03:28:03.927 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:03.702+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:03.928 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:03.702+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:03.928 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:03.702+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:03.928 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:03.702+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:04.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:03.974+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:04.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:03.974+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:04.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:03.974+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:04.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:03 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:03.974+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:04.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:04.006+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:04.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:04.006+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:04.191 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:04.006+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:04.191 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:04.006+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:04.827 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:28:04.828 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:28:04.894 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:28:04.894 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-06T03:28:04.895 INFO:tasks.cephadm:Removing cluster... 2024-04-06T03:28:04.895 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 --force 2024-04-06T03:28:04.999 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:04.747+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:04.999 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:04.747+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:04.999 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:04.747+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:04.999 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:04.747+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:05.053 INFO:teuthology.orchestra.run.smithi105.stdout:Deleting cluster with fsid: 130df578-f3bf-11ee-b649-cb9ed24678a4 2024-04-06T03:28:05.422 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:28:05 smithi105 systemd[1]: Stopping Ceph mon.smithi105 for 130df578-f3bf-11ee-b649-cb9ed24678a4... 2024-04-06T03:28:05.423 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:04.998+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:05.423 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:04.998+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:05.423 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:04 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:04.998+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:05.423 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:04.999+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:05.423 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:05.005+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:05.423 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:05.005+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:05.424 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:05.005+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:05.424 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:05.005+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:05.689 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:28:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:28:05.422+0000 7fd04be0f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi105 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T03:28:05.689 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:28:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105[29224]: 2024-04-06T03:28:05.422+0000 7fd04be0f700 -1 mon.smithi105@0(probing) e2 *** Got Signal Terminated *** 2024-04-06T03:28:06.040 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:28:05 smithi105 podman[85072]: 2024-04-06 03:28:05.769501092 +0000 UTC m=+0.487095387 container died d27bf0498159c5eb042ec29dc651c2d3cde19f6c07dbc359b8c9870c0e665bda (image=quay.io/ceph/ceph:v18.2.1, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.29.1) 2024-04-06T03:28:06.040 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:05.986+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:06.040 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:05.986+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:06.040 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:05.986+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:06.040 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:05.987+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:06.041 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:05.763+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:06.041 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:05.763+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:06.041 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:05.763+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:06.041 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:05 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:05.763+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:06.374 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:28:06 smithi105 podman[85072]: 2024-04-06 03:28:06.157157034 +0000 UTC m=+0.874751339 container cleanup d27bf0498159c5eb042ec29dc651c2d3cde19f6c07dbc359b8c9870c0e665bda (image=quay.io/ceph/ceph:v18.2.1, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, ceph=True, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-06T03:28:06.375 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:28:06 smithi105 bash[85072]: ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105 2024-04-06T03:28:06.375 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:06.039+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:06.375 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:06.039+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:06.375 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:06.039+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:06.375 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:06.040+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:06.689 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:28:06 smithi105 podman[85086]: 2024-04-06 03:28:06.375662196 +0000 UTC m=+0.601664758 container remove d27bf0498159c5eb042ec29dc651c2d3cde19f6c07dbc359b8c9870c0e665bda (image=quay.io/ceph/ceph:v18.2.1, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-mon-smithi105, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240222) 2024-04-06T03:28:06.690 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:28:06 smithi105 systemd[1]: ceph-130df578-f3bf-11ee-b649-cb9ed24678a4@mon.smithi105.service: Deactivated successfully. 2024-04-06T03:28:06.690 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:28:06 smithi105 systemd[1]: Stopped Ceph mon.smithi105 for 130df578-f3bf-11ee-b649-cb9ed24678a4. 2024-04-06T03:28:06.690 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Apr 06 03:28:06 smithi105 systemd[1]: ceph-130df578-f3bf-11ee-b649-cb9ed24678a4@mon.smithi105.service: Consumed 29.995s CPU time. 2024-04-06T03:28:06.978 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:06.719+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:06.979 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:06.719+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:06.979 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:06.719+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:06.979 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:06 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:06.719+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:07.296 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:07.026+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:07.296 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:07.027+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:07.296 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:07.027+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:07.296 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:07.027+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:07.296 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:07.043+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:07.297 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:07.043+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:07.297 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:07.043+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:07.297 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:07.043+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:07.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:07.689+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:07.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:07.689+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:07.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:07.689+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:07.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:07.689+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:08.236 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:07.984+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:08.236 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:07.984+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:08.236 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:07.984+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:08.236 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:07 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:07.985+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:08.237 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:08.020+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:08.237 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:08.021+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:08.237 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:08.021+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:08.237 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:08.021+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:08.894 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:08.704+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:08.894 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:08.704+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:08.894 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:08.704+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:08.894 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:08 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:08.704+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:09.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:09.016+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:09.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:09.016+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:09.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:09.016+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:09.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:09.016+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:09.191 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:09.005+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:09.191 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:09.005+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:09.191 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:09.005+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:09.191 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:09.005+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:09.814 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:09.679+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:09.815 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:09.679+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:09.815 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:09.679+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:09.815 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:09 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:09.680+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:10.165 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:10.028+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:10.165 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:10.028+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:10.165 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:10.028+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:10.165 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:10.028+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:10.166 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:10.021+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:10.166 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:10.021+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:10.166 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:10.021+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:10.166 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:10.021+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:10.804 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:10.664+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:10.804 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:10.664+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:10.805 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:10.664+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:10.805 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:10 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:10.665+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:11.069 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:11.032+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:11.069 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:11.032+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:11.070 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:11.032+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:11.070 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:11.032+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:11.070 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:11.042+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:11.070 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:11.042+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:11.070 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:11.042+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:11.071 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:11.042+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:11.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:11.648+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:11.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:11.648+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:11.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:11.648+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:11.690 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:11 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:11.648+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:12.358 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:12.037+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:12.358 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:12.037+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:12.358 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:12.037+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:12.359 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:12.038+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:12.359 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:12.041+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:12.359 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:12.041+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:12.359 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:12.041+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:12.359 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:12.041+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:13.071 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:12.692+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:13.071 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:12.692+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:13.071 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:12.692+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:13.071 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:12 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:12.692+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:13.389 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:13.073+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:13.389 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:13.073+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:13.390 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:13.073+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:13.390 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:13.073+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:13.390 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:13.070+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:13.390 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:13.070+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:13.390 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:13.070+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:13.390 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:13.070+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:13.939 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:13.653+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:13.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:13.653+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:13.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:13.653+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:13.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:13 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:13.653+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:14.375 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:14.100+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:14.375 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:14.100+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:14.375 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:14.100+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:14.376 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:14.100+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:14.376 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:14.039+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:14.376 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:14.039+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:14.376 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:14.039+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:14.376 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:14.039+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:14.902 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:14.699+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:14.902 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:14.699+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:14.902 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:14.699+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:14.902 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:14 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:14.699+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:15.189 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:15.124+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:15.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:15.124+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:15.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:15.124+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:15.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:15.124+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:15.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:15.045+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:15.191 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:15.045+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:15.191 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:15.045+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:15.191 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:15.045+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:15.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:15 smithi105 systemd[1]: Stopping Ceph osd.0 for 130df578-f3bf-11ee-b649-cb9ed24678a4... 2024-04-06T03:28:15.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:15.738+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:15.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:15.738+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:15.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:15.738+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:15.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:15.739+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:15.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:15.880+0000 7f11c376e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T03:28:15.941 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:15.880+0000 7f11c376e700 -1 osd.0 47 *** Got signal Terminated *** 2024-04-06T03:28:15.941 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:15 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:15.880+0000 7f11c376e700 -1 osd.0 47 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:28:16.439 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:16.010+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:16.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:16.010+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:16.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:16.010+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:16.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:16.010+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:16.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:16.153+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:16.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:16.153+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:16.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:16.153+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:16.441 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:16.153+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:17.001 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:16.734+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:17.002 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:16.734+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:17.002 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:16.734+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:17.002 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:16 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:16.734+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:17.252 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:17.001+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:17.253 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:17.001+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:17.253 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:17.001+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:17.253 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:17.001+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:17.253 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:17.106+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:17.253 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:17.106+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:17.253 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:17.106+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:17.254 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:17.106+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:18.125 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:17.741+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:18.125 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:17.741+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:18.126 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:17.741+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:18.126 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:17.741+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:18.126 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:17.952+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:18.126 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:17.952+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:18.126 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:17.952+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:18.126 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:17 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:17.953+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:18.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:18.125+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:18.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:18.125+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:18.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:18.125+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:18.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:18.125+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:18.983 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:18.711+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:18.983 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:18.711+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:18.983 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:18.711+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:18.983 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:18.711+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:19.442 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:19.144+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:19.442 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:19.144+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:19.442 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:19.144+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:19.443 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:19.144+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:19.443 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:18.983+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:19.443 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:18.983+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:19.443 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:18.983+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:19.443 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:18 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:18.983+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:20.172 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:19.936+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:20.172 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:19.936+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:20.173 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:19.936+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:20.173 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:19.936+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:20.173 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:19.700+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:20.173 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:19.700+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:20.173 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:19.700+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:20.173 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:19 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:19.700+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:20.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:20.172+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:20.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:20.172+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:20.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:20.172+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:20.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:20.172+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:20.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:20.674+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:20.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:20.674+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:20.932 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:20.674+0000 7f11c0565700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:42.275324+0000 (oldest deadline 2024-04-06T03:23:02.275324+0000) 2024-04-06T03:28:20.933 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0[47546]: 2024-04-06T03:28:20.674+0000 7f11c0565700 -1 osd.0 47 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.24297.0:10218 3.1 3:806f16f0:::10000004206.00000000:head [write 0~13440 [1@-1] in=13440b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:21.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:20.932+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:21.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:20.932+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:21.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:20.932+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:21.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:20 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:20.932+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:21.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:21.149+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:21.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:21.149+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:21.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:21.149+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:21.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:21.149+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:21.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:21 smithi105 podman[86038]: 2024-04-06 03:28:21.365724302 +0000 UTC m=+5.576633578 container died 339c3f865ef940d1aacc7582ee7a5dab7a73eda087ea8c10f6e4d6ad0279d353 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222) 2024-04-06T03:28:21.689 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:21 smithi105 podman[86038]: 2024-04-06 03:28:21.689549227 +0000 UTC m=+5.900458510 container cleanup 339c3f865ef940d1aacc7582ee7a5dab7a73eda087ea8c10f6e4d6ad0279d353 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True) 2024-04-06T03:28:21.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:21 smithi105 bash[86038]: ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0 2024-04-06T03:28:21.940 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:21 smithi105 podman[86059]: 2024-04-06 03:28:21.924696258 +0000 UTC m=+0.555248968 container remove 339c3f865ef940d1aacc7582ee7a5dab7a73eda087ea8c10f6e4d6ad0279d353 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1) 2024-04-06T03:28:21.941 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:21.914+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:21.941 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:21.914+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:21.941 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:21.914+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:21.942 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:21 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:21.914+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:22.439 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:22 smithi105 podman[86154]: 2024-04-06 03:28:22.253304869 +0000 UTC m=+0.017975817 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3 2024-04-06T03:28:22.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:22.126+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:22.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:22.126+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:22.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:22.126+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:22.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:22.127+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:22.736 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:22 smithi105 podman[86154]: 2024-04-06 03:28:22.526018615 +0000 UTC m=+0.290689539 container create baea65e27fcad6b1dddfca4dfa2ed530828bc5dfbecb0887cb3b480504c342d2 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.vendor=CentOS, RELEASE=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-06T03:28:23.030 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:22.877+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:23.030 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:22.877+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:23.030 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:22.877+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:23.030 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:22 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:22.878+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:23.030 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:22 smithi105 podman[86154]: 2024-04-06 03:28:22.736578055 +0000 UTC m=+0.501248980 container init baea65e27fcad6b1dddfca4dfa2ed530828bc5dfbecb0887cb3b480504c342d2 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-04-06T03:28:23.031 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:22 smithi105 podman[86154]: 2024-04-06 03:28:22.739837377 +0000 UTC m=+0.504508304 container start baea65e27fcad6b1dddfca4dfa2ed530828bc5dfbecb0887cb3b480504c342d2 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0) 2024-04-06T03:28:23.031 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:22 smithi105 podman[86154]: 2024-04-06 03:28:22.828502435 +0000 UTC m=+0.593173389 container attach baea65e27fcad6b1dddfca4dfa2ed530828bc5dfbecb0887cb3b480504c342d2 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, ceph=True, GIT_CLEAN=True) 2024-04-06T03:28:23.031 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:23 smithi105 podman[86154]: 2024-04-06 03:28:23.030659358 +0000 UTC m=+0.795330300 container died baea65e27fcad6b1dddfca4dfa2ed530828bc5dfbecb0887cb3b480504c342d2 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-06T03:28:23.293 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:23.082+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:23.293 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:23.082+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:23.293 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:23.082+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:23.293 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:23.083+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:23.901 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:23 smithi105 podman[86179]: 2024-04-06 03:28:23.649388918 +0000 UTC m=+0.613530976 container remove baea65e27fcad6b1dddfca4dfa2ed530828bc5dfbecb0887cb3b480504c342d2 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-0-deactivate, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, RELEASE=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-06T03:28:23.902 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:23 smithi105 systemd[1]: ceph-130df578-f3bf-11ee-b649-cb9ed24678a4@osd.0.service: Deactivated successfully. 2024-04-06T03:28:23.902 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:23 smithi105 systemd[1]: Stopped Ceph osd.0 for 130df578-f3bf-11ee-b649-cb9ed24678a4. 2024-04-06T03:28:23.902 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 06 03:28:23 smithi105 systemd[1]: ceph-130df578-f3bf-11ee-b649-cb9ed24678a4@osd.0.service: Consumed 27.670s CPU time. 2024-04-06T03:28:24.161 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:23 smithi105 systemd[1]: Stopping Ceph osd.1 for 130df578-f3bf-11ee-b649-cb9ed24678a4... 2024-04-06T03:28:24.161 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:23.920+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:24.161 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:23.920+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:24.161 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:23.920+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:24.161 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:23 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:23.921+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:24.161 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:24.160+0000 7fdfe73b9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T03:28:24.162 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:24.160+0000 7fdfe73b9700 -1 osd.1 47 *** Got signal Terminated *** 2024-04-06T03:28:24.162 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:24.160+0000 7fdfe73b9700 -1 osd.1 47 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:28:24.162 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:24.039+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:24.162 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:24.039+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:24.162 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:24.039+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:24.162 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:24.040+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:25.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:24.954+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:25.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:24.954+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:25.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:24.954+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:25.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:24.954+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:25.441 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:24.994+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:25.441 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:24.994+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:25.441 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:24.994+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:25.441 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:24 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:24.994+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:26.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:25.974+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:26.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:25.974+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:26.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:25.974+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:26.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:25 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:25.974+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:26.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:25.999+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:26.441 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:25.999+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:26.441 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:25.999+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:26.441 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:25.999+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:27.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:27.010+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:27.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:27.010+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:27.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:27.010+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:27.441 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:27.010+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:27.441 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:26.991+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:27.441 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:26.991+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:27.441 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:26.991+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:27.441 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:26 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:26.991+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:28.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:28.053+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:28.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:28.053+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:28.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:28.053+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:28.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:28.053+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:28.441 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:27.973+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:28.441 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:27.973+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:28.441 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:27.973+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:28.441 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:27 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:27.973+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:29.439 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:28.960+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:29.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:28.960+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:29.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:28.960+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:29.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:28 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:28.960+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:29.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:29.045+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:29.440 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:29.045+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:29.441 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:29.045+0000 7fdfe01a8700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:38.697544+0000 (oldest deadline 2024-04-06T03:22:58.697544+0000) 2024-04-06T03:28:29.441 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:29 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1[55303]: 2024-04-06T03:28:29.045+0000 7fdfe01a8700 -1 osd.1 47 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.11:581 2.d 2:b02ffd62:::200.00000010:head [write 2110431~9017 [fadvise_dontneed] in=9017b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-04-06T03:28:29.911 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:29 smithi105 podman[86267]: 2024-04-06 03:28:29.641204664 +0000 UTC m=+5.587490240 container died eb4cf9ea6b1bcfe454cd6f1c83b66e567d37f27a517257f4f9145c6e92990a70 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1) 2024-04-06T03:28:30.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:29 smithi105 podman[86267]: 2024-04-06 03:28:29.994529266 +0000 UTC m=+5.940814850 container cleanup eb4cf9ea6b1bcfe454cd6f1c83b66e567d37f27a517257f4f9145c6e92990a70 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.29.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.build-date=20240222) 2024-04-06T03:28:30.190 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:29 smithi105 bash[86267]: ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1 2024-04-06T03:28:30.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:30.009+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:30.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:30.009+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:30.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:30.009+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:30.191 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:30 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:30.010+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:30.611 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:30 smithi105 podman[86286]: 2024-04-06 03:28:30.296223065 +0000 UTC m=+0.650588803 container remove eb4cf9ea6b1bcfe454cd6f1c83b66e567d37f27a517257f4f9145c6e92990a70 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-04-06T03:28:30.924 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:30 smithi105 podman[86379]: 2024-04-06 03:28:30.514494159 +0000 UTC m=+0.017814373 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3 2024-04-06T03:28:30.924 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:30 smithi105 podman[86379]: 2024-04-06 03:28:30.630252833 +0000 UTC m=+0.133573034 container create 4a2620364cf8f4d630c25e7a901066a79c3186d86e37521f194d127708df7692 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1-deactivate, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-06T03:28:30.924 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:30 smithi105 podman[86379]: 2024-04-06 03:28:30.84122968 +0000 UTC m=+0.344549880 container init 4a2620364cf8f4d630c25e7a901066a79c3186d86e37521f194d127708df7692 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1-deactivate, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD) 2024-04-06T03:28:30.924 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:30 smithi105 podman[86379]: 2024-04-06 03:28:30.844373611 +0000 UTC m=+0.347693821 container start 4a2620364cf8f4d630c25e7a901066a79c3186d86e37521f194d127708df7692 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1-deactivate, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-04-06T03:28:30.924 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:30 smithi105 podman[86379]: 2024-04-06 03:28:30.924503178 +0000 UTC m=+0.427823387 container attach 4a2620364cf8f4d630c25e7a901066a79c3186d86e37521f194d127708df7692 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1-deactivate, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0) 2024-04-06T03:28:31.189 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:31 smithi105 podman[86379]: 2024-04-06 03:28:31.125358461 +0000 UTC m=+0.628678662 container died 4a2620364cf8f4d630c25e7a901066a79c3186d86e37521f194d127708df7692 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux ) 2024-04-06T03:28:31.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:31.035+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:31.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:31.035+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:31.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:31.035+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:31.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:31 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:31.035+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:32.043 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:31 smithi105 systemd[1]: Stopping Ceph osd.2 for 130df578-f3bf-11ee-b649-cb9ed24678a4... 2024-04-06T03:28:32.043 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:31 smithi105 podman[86404]: 2024-04-06 03:28:31.703080608 +0000 UTC m=+0.572357447 container remove 4a2620364cf8f4d630c25e7a901066a79c3186d86e37521f194d127708df7692 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-1-deactivate, ceph=True, RELEASE=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) 2024-04-06T03:28:32.044 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:31 smithi105 systemd[1]: ceph-130df578-f3bf-11ee-b649-cb9ed24678a4@osd.1.service: Deactivated successfully. 2024-04-06T03:28:32.044 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:31 smithi105 systemd[1]: Stopped Ceph osd.1 for 130df578-f3bf-11ee-b649-cb9ed24678a4. 2024-04-06T03:28:32.044 INFO:journalctl@ceph.osd.1.smithi105.stdout:Apr 06 03:28:31 smithi105 systemd[1]: ceph-130df578-f3bf-11ee-b649-cb9ed24678a4@osd.1.service: Consumed 28.805s CPU time. 2024-04-06T03:28:32.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:32.042+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:32.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:32.042+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:32.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:32.042+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:32.441 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:32.042+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:32.441 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:32.146+0000 7fb163c7c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-06T03:28:32.441 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:32.146+0000 7fb163c7c700 -1 osd.2 47 *** Got signal Terminated *** 2024-04-06T03:28:32.441 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:32 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:32.146+0000 7fb163c7c700 -1 osd.2 47 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:28:33.439 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:33.050+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:33.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:33.050+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:33.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:33.050+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:33.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:33 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:33.050+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:34.339 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:34.012+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:34.339 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:34.012+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:34.339 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:34.012+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:34.339 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:34.012+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:35.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:34.970+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:35.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:34.970+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:35.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:34.970+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:35.440 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:34 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:34.970+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:36.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:35.929+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:36.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:35.929+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:36.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:35.929+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:36.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:35 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:35.929+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:37.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:36.927+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6804 osd.3 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:37.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:36.927+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6812 osd.4 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:37.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:36.927+0000 7fb15ea6f700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.184:6820 osd.5 ever on either front or back, first ping sent 2024-04-06T03:22:40.000187+0000 (oldest deadline 2024-04-06T03:23:00.000187+0000) 2024-04-06T03:28:37.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:36 smithi105 ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2[62684]: 2024-04-06T03:28:36.927+0000 7fb15ea6f700 -1 osd.2 47 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.24297.0:10225 3.7 3:e3cfe7dc:::1000000420b.00000000:head [write 0~2289 [1@-1] in=2289b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-04-06T03:28:37.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:37 smithi105 podman[86481]: 2024-04-06 03:28:37.624276401 +0000 UTC m=+5.579240927 container died 0392f09bdb6bda7e5e49fd7be41f18ac64e9b8baa23f90140dbcc9ef586ffca4 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-04-06T03:28:37.938 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:37 smithi105 podman[86481]: 2024-04-06 03:28:37.939026308 +0000 UTC m=+5.893990834 container cleanup 0392f09bdb6bda7e5e49fd7be41f18ac64e9b8baa23f90140dbcc9ef586ffca4 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1) 2024-04-06T03:28:38.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:37 smithi105 bash[86481]: ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2 2024-04-06T03:28:38.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:38 smithi105 podman[86499]: 2024-04-06 03:28:38.182392874 +0000 UTC m=+0.555386045 container remove 0392f09bdb6bda7e5e49fd7be41f18ac64e9b8baa23f90140dbcc9ef586ffca4 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-06T03:28:38.902 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:38 smithi105 podman[86588]: 2024-04-06 03:28:38.509329704 +0000 UTC m=+0.018220423 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3 2024-04-06T03:28:38.902 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:38 smithi105 podman[86588]: 2024-04-06 03:28:38.624817103 +0000 UTC m=+0.133707809 container create 78a4a1e934ec162a3eeb4b33595d95ce5ab8fa21ec3e880d6b9200e9168a9f08 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1) 2024-04-06T03:28:38.902 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:38 smithi105 podman[86588]: 2024-04-06 03:28:38.8280399 +0000 UTC m=+0.336930621 container init 78a4a1e934ec162a3eeb4b33595d95ce5ab8fa21ec3e880d6b9200e9168a9f08 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2-deactivate, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-04-06T03:28:38.902 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:38 smithi105 podman[86588]: 2024-04-06 03:28:38.831335257 +0000 UTC m=+0.340225966 container start 78a4a1e934ec162a3eeb4b33595d95ce5ab8fa21ec3e880d6b9200e9168a9f08 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0) 2024-04-06T03:28:38.903 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:38 smithi105 podman[86588]: 2024-04-06 03:28:38.902514741 +0000 UTC m=+0.411405483 container attach 78a4a1e934ec162a3eeb4b33595d95ce5ab8fa21ec3e880d6b9200e9168a9f08 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2-deactivate, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2) 2024-04-06T03:28:39.190 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:39 smithi105 podman[86588]: 2024-04-06 03:28:39.108909629 +0000 UTC m=+0.617800330 container died 78a4a1e934ec162a3eeb4b33595d95ce5ab8fa21ec3e880d6b9200e9168a9f08 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True) 2024-04-06T03:28:39.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:39 smithi105 podman[86616]: 2024-04-06 03:28:39.655905246 +0000 UTC m=+0.538170050 container remove 78a4a1e934ec162a3eeb4b33595d95ce5ab8fa21ec3e880d6b9200e9168a9f08 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-130df578-f3bf-11ee-b649-cb9ed24678a4-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, ceph=True) 2024-04-06T03:28:39.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:39 smithi105 systemd[1]: ceph-130df578-f3bf-11ee-b649-cb9ed24678a4@osd.2.service: Deactivated successfully. 2024-04-06T03:28:39.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:39 smithi105 systemd[1]: Stopped Ceph osd.2 for 130df578-f3bf-11ee-b649-cb9ed24678a4. 2024-04-06T03:28:39.940 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 06 03:28:39 smithi105 systemd[1]: ceph-130df578-f3bf-11ee-b649-cb9ed24678a4@osd.2.service: Consumed 25.613s CPU time. 2024-04-06T03:28:45.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:28:45.280 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:28:48.350 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:28:57.352 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:28:57.353 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:29:00.418 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:29:12.420 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:29:12.420 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:29:15.490 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:29:18.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:29:18.492 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:29:18.562 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:29:18.563 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-06T03:29:18.563 INFO:tasks.cephadm:Teardown complete 2024-04-06T03:29:18.563 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/cephadm.py", line 1553, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/cephadm.py", line 1810, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/cephadm.py", line 1686, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/cephadm.py", line 142, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/cephadm.py", line 310, in _rm_cluster ctx.cluster.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi184 2024-04-06T03:29:18.587 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-06T03:29:18.599 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-04-06T03:29:18.616 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-06T03:29:18.616 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-06T03:29:18.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:29:18.629 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:29:21.694 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:29:30.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:29:30.697 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:29:33.758 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:29:45.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:29:45.760 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:29:48.830 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:29:51.831 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:29:51.832 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:29:51.902 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:29:51.902 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-06T03:29:51.953 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi184 2024-04-06T03:29:51.974 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-06T03:29:51.986 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-06T03:29:51.986 DEBUG:teuthology.orchestra.run.smithi105:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T03:29:51.994 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:29:51.995 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:29:52.009 INFO:teuthology.orchestra.run.smithi105.stderr:bash: line 1: ntpq: command not found 2024-04-06T03:29:52.014 INFO:teuthology.orchestra.run.smithi105.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-06T03:29:52.014 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================== 2024-04-06T03:29:52.014 INFO:teuthology.orchestra.run.smithi105.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 2 -454us[ -454us] +/- 80ms 2024-04-06T03:29:52.014 INFO:teuthology.orchestra.run.smithi105.stdout:^* hv02.front.sepia.ceph.com 3 7 377 68 -123us[ -117us] +/- 56ms 2024-04-06T03:29:52.014 INFO:teuthology.orchestra.run.smithi105.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 +575us[ +575us] +/- 73ms 2024-04-06T03:29:52.015 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-06T03:29:55.070 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:30:04.072 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:30:04.073 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:30:07.134 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:30:19.136 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:30:19.136 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:30:22.206 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:30:25.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:30:25.208 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:30:25.278 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:30:25.278 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-06T03:30:25.279 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi184 2024-04-06T03:30:25.294 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-06T03:30:25.304 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-06T03:30:25.305 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-06T03:30:25.336 DEBUG:teuthology.orchestra.run.smithi105:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-06T03:30:25.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:30:25.425 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:30:28.478 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:30:37.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:30:37.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:30:40.546 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:30:52.547 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:30:52.548 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:30:55.618 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:30:58.620 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:30:58.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:30:58.686 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:30:58.686 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-06T03:30:58.687 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi184 2024-04-06T03:30:58.695 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-06T03:30:58.706 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-06T03:30:58.750 INFO:teuthology.task.internal:Duration was 3437.534482 seconds 2024-04-06T03:30:58.751 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-06T03:30:58.775 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-06T03:30:58.776 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-06T03:30:58.782 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:30:58.782 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:30:58.829 INFO:teuthology.orchestra.run.smithi105.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-06T03:31:01.854 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:31:10.856 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:31:10.856 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:31:13.918 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:31:25.919 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:31:25.921 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:31:28.994 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:31:31.996 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:31:31.996 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:31:32.062 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:31:32.062 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-06T03:31:32.063 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi184 2024-04-06T03:31:32.078 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-06T03:31:32.090 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-06T03:31:32.090 DEBUG:teuthology.orchestra.run.smithi105:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-06T03:31:32.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:31:32.127 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:31:35.198 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:31:44.199 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:31:44.200 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:31:47.262 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:31:59.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:31:59.264 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:32:02.334 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:32:05.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:32:05.336 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:32:05.406 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:32:05.406 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-06T03:32:05.407 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi184 2024-04-06T03:32:05.421 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-06T03:32:05.434 DEBUG:teuthology.orchestra.run.smithi105:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-06T03:32:05.440 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:32:05.441 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:32:05.468 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = core 2024-04-06T03:32:08.510 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:32:17.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:32:17.513 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:32:20.574 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:32:32.576 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:32:32.577 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:32:35.646 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:32:38.647 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:32:38.648 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:32:38.718 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:32:38.718 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-06T03:32:38.719 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi184 2024-04-06T03:32:38.733 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-06T03:32:38.749 INFO:teuthology.task.internal:Transferring archived files... 2024-04-06T03:32:38.751 DEBUG:teuthology.misc:Transferring archived files from smithi105:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641871/remote/smithi105 2024-04-06T03:32:38.751 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-06T03:32:38.794 DEBUG:teuthology.misc:Transferring archived files from smithi184:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641871/remote/smithi184 2024-04-06T03:32:38.795 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:32:38.795 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:32:41.854 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:32:50.856 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:32:50.857 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:32:53.922 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:33:05.923 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:33:05.924 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:33:08.990 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:33:11.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:33:11.992 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:33:12.062 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:33:12.062 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-06T03:33:12.063 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi184 2024-04-06T03:33:12.080 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-06T03:33:12.100 INFO:teuthology.task.internal:Not uploading archives. 2024-04-06T03:33:12.100 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-06T03:33:12.135 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-06T03:33:12.136 DEBUG:teuthology.orchestra.run.smithi105:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-06T03:33:12.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:33:12.144 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:33:12.158 INFO:teuthology.orchestra.run.smithi105.stdout: 265306 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 6 03:29 /home/ubuntu/cephtest 2024-04-06T03:33:12.159 INFO:teuthology.orchestra.run.smithi105.stdout: 265307 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Apr 6 03:32 /home/ubuntu/cephtest/archive 2024-04-06T03:33:12.159 INFO:teuthology.orchestra.run.smithi105.stdout: 265311 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 6 02:33 /home/ubuntu/cephtest/archive/syslog 2024-04-06T03:33:12.159 INFO:teuthology.orchestra.run.smithi105.stdout: 265313 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Apr 6 02:33 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-06T03:33:12.159 INFO:teuthology.orchestra.run.smithi105.stdout: 265312 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Apr 6 02:33 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-06T03:33:12.159 INFO:teuthology.orchestra.run.smithi105.stdout: 655361 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 6 03:30 /home/ubuntu/cephtest/archive/audit 2024-04-06T03:33:12.159 INFO:teuthology.orchestra.run.smithi105.stdout: 655363 176 -rw------- 1 ubuntu ubuntu 179708 Apr 6 03:30 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-04-06T03:33:12.159 INFO:teuthology.orchestra.run.smithi105.stdout: 265396 212 -rwxr-xr-x 1 ubuntu ubuntu 215334 Apr 6 02:40 /home/ubuntu/cephtest/cephadm 2024-04-06T03:33:12.160 INFO:teuthology.orchestra.run.smithi105.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-06T03:33:15.202 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:33:24.203 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:33:24.204 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:33:27.262 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:33:39.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:33:39.302 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:33:42.366 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:33:45.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-04-06T03:33:45.368 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:33:45.438 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-04-06T03:33:45.438 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-04-06T03:33:45.439 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi184 2024-04-06T03:33:45.454 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-06T03:33:45.469 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-06T03:33:45.524 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: OSError: [Errno 113] No route to host 2024-04-06T03:33:45.524 INFO:teuthology.nuke:Checking targets against current locks 2024-04-06T03:33:45.556 DEBUG:teuthology.nuke:shortname: smithi105 2024-04-06T03:33:45.556 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-06T03:33:45.579 DEBUG:teuthology.nuke:shortname: smithi184 2024-04-06T03:33:45.579 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-06T03:33:45.582 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641871', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 02:24:48.433530', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEnjEFQecBD4YhMZ8ECB/t6x08y1pkUu7IBFqwpC+gEC03+dZViMj512rGlYNPFNtbHGpo7SJr0HmMGsVW7ofq8='} 2024-04-06T03:33:45.607 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641871', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 02:24:48.432059', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA3MvAVm7agvBv8Rfka6x0uG9XpxK6I5pmaz0CfxKt2UPPrNCDKEyzauHb95RoM0tF1IyLwDyntySDo7GwNpQn0='} 2024-04-06T03:33:45.636 INFO:teuthology.orchestra.console.smithi105:Power off 2024-04-06T03:33:45.636 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-06T03:33:45.659 DEBUG:teuthology.orchestra.console.smithi105:power off output: Chassis Power Control: Down/Off 2024-04-06T03:33:45.659 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T03:33:45.674 INFO:teuthology.orchestra.console.smithi184:Power off 2024-04-06T03:33:45.675 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-06T03:33:45.684 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on 2024-04-06T03:33:45.697 DEBUG:teuthology.orchestra.console.smithi184:power off output: Chassis Power Control: Down/Off 2024-04-06T03:33:45.697 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T03:33:45.719 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is on 2024-04-06T03:33:49.685 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T03:33:49.721 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T03:33:49.795 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on 2024-04-06T03:33:49.831 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is on 2024-04-06T03:33:53.800 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T03:33:53.832 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T03:33:53.912 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on 2024-04-06T03:33:53.942 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is on 2024-04-06T03:33:57.916 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T03:33:57.943 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T03:33:58.026 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is off 2024-04-06T03:33:58.053 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is off 2024-04-06T03:33:58.128 INFO:teuthology.orchestra.console.smithi105:Power off completed 2024-04-06T03:33:58.155 INFO:teuthology.orchestra.console.smithi184:Power off completed 2024-04-06T03:33:58.261 INFO:teuthology.run:Summary data: description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 3437.5344817638397 failure_reason: 'Command failed on smithi105 with status 1: ''sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 130df578-f3bf-11ee-b649-cb9ed24678a4 -e sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -- bash -c \''ceph versions | jq -e \''"\''"\''.mgr | keys\''"\''"\'' | grep $sha1\''''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=14ac10fa5df845a0a9e4ccf6db7a87c5 status: fail success: false 2024-04-06T03:33:58.261 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T03:33:58.429 INFO:teuthology.run:FAIL